Note: "permalinks" may not be as permanent as we would like,
direct links of old sources may well be a few messages off.
Memo to Self: Self, try Reply-All if you want more than one person to look at your problem. ;-) Original message: Ok. I have six hours of log data, but it all says pretty much the same thing. Here's the first 15 minutes or so. If you want more, I'll send more. Thanks for looking at this. -- Regards, David A. Buechler System Administrator, CableAmerica Missouri /var/log/messages: Sep 15 04:17:38 smtp2 heartbeat: [2717]: WARN: Gmain_timeout_dispatch: Dispatch function for send local status took too long to execute: 150 ms (> 50 ms) (GSource: 0x1aacc9f8) Sep 15 04:19:38 smtp2 snmpd[2489]: Connection from UDP: [206.16.46.12]:46136 Sep 15 04:19:38 smtp2 snmpd[2489]: Received SNMP packet(s) from UDP: [206.16.46.12]:46136 Sep 15 04:19:42 smtp2 snmpd[2489]: Connection from UDP: [206.16.46.12]:46136 Sep 15 04:19:49 smtp2 last message repeated 35 times Sep 15 04:31:41 smtp2 kernel: block drbd0: sock was shut down by peer Sep 15 04:31:42 smtp2 kernel: block drbd0: peer( Primary -> Unknown ) conn( Connected -> BrokenPipe ) pdsk( UpToDate -> DUnknown ) Sep 15 04:31:42 smtp2 kernel: block drbd0: short read expecting header on sock: r=0 Sep 15 04:31:42 smtp2 kernel: block drbd0: md_sync_timer expired! Worker calls drbd_md_sync(). Sep 15 04:31:42 smtp2 kernel: block drbd0: msock_sendmsg returned -32 Sep 15 04:31:42 smtp2 kernel: block drbd0: short sent PingAck size=8 sent=0 Sep 15 04:31:42 smtp2 kernel: block drbd0: asender terminated Sep 15 04:31:42 smtp2 kernel: block drbd0: Terminating asender thread Sep 15 04:31:42 smtp2 kernel: block drbd0: Connection closed Sep 15 04:31:42 smtp2 kernel: block drbd0: conn( BrokenPipe -> Unconnected ) Sep 15 04:31:42 smtp2 kernel: block drbd0: receiver terminated Sep 15 04:31:42 smtp2 kernel: block drbd0: Restarting receiver thread Sep 15 04:31:42 smtp2 kernel: block drbd0: receiver (re)started Sep 15 04:31:42 smtp2 kernel: block drbd0: conn( Unconnected -> WFConnection ) Sep 15 04:31:42 smtp2 kernel: block drbd0: Handshake successful: Agreed network protocol version 90 Sep 15 04:31:42 smtp2 kernel: block drbd0: conn( WFConnection -> WFReportParams ) Sep 15 04:31:42 smtp2 kernel: block drbd0: Starting asender thread (from drbd0_receiver [2581]) Sep 15 04:31:42 smtp2 kernel: block drbd0: data-integrity-alg: <not-used> Sep 15 04:31:42 smtp2 kernel: block drbd0: drbd_sync_handshake: Sep 15 04:31:42 smtp2 kernel: block drbd0: self E6B7DB61310BE3A6:0000000000000000:B939FB24AD00D552:7E8DF909A8E37EDF bits:0 flags:0 Sep 15 04:31:42 smtp2 kernel: block drbd0: peer E720051EDAD32591:E6B7DB61310BE3A7:B939FB24AD00D552:7E8DF909A8E37EDF bits:1090 flags:0 Sep 15 04:31:42 smtp2 kernel: block drbd0: uuid_compare()=-1 by rule 5 Sep 15 04:31:42 smtp2 kernel: block drbd0: peer( Unknown -> Primary ) conn( WFReportParams -> WFBitMapT ) pdsk( DUnknown -> UpToDate ) Sep 15 04:31:42 smtp2 kernel: block drbd0: conn( WFBitMapT -> WFSyncUUID ) Sep 15 04:31:42 smtp2 kernel: block drbd0: helper command: /sbin/drbdadm before-resync-target minor-0 Sep 15 04:31:42 smtp2 kernel: block drbd0: md_sync_timer expired! Worker calls drbd_md_sync(). Sep 15 04:31:42 smtp2 kernel: block drbd0: helper command: /sbin/drbdadm before-resync-target minor-0 exit code 0 (0x0) Sep 15 04:31:42 smtp2 kernel: block drbd0: conn( WFSyncUUID -> SyncTarget ) disk( UpToDate -> Inconsistent ) Sep 15 04:31:42 smtp2 kernel: block drbd0: Began resync as SyncTarget (will sync 4360 KB [1090 bits set]). Sep 15 04:31:42 smtp2 kernel: block drbd0: Resync done (total 3 sec; paused 0 sec; 1452 K/sec) Sep 15 04:31:42 smtp2 kernel: block drbd0: conn( SyncTarget -> Connected ) disk( Inconsistent -> UpToDate ) Sep 15 04:31:42 smtp2 kernel: block drbd0: helper command: /sbin/drbdadm after-resync-target minor-0 Sep 15 04:31:42 smtp2 kernel: drbdadm invoked oom-killer: gfp_mask=0x84d0, order=0, oomkilladj=0 Sep 15 04:31:42 smtp2 kernel: Sep 15 04:31:42 smtp2 kernel: Call Trace: Sep 15 04:31:42 smtp2 kernel: [<ffffffff800c3bda>] out_of_memory+0x8e/0x2f3 Sep 15 04:31:42 smtp2 kernel: [<ffffffff8000f2ea>] __alloc_pages+0x245/0x2ce Sep 15 04:31:42 smtp2 kernel: [<ffffffff8002309f>] alloc_page_interleave+0x3d/0x74 Sep 15 04:31:42 smtp2 kernel: [<ffffffff8002b5e5>] get_zeroed_page+0x21/0x82 Sep 15 04:31:42 smtp2 kernel: [<ffffffff8002d80c>] __pmd_alloc+0x14/0x8c Sep 15 04:31:42 smtp2 kernel: [<ffffffff80008127>] copy_page_range+0x1f3/0x73e Sep 15 04:31:42 smtp2 kernel: [<ffffffff800d6f0e>] alternate_node_alloc+0x70/0x8c Sep 15 04:31:42 smtp2 kernel: [<ffffffff8001f847>] copy_process+0xd30/0x15b8 Sep 15 04:31:42 smtp2 kernel: [<ffffffff80030d3a>] do_fork+0x69/0x1c1 Sep 15 04:31:42 smtp2 kernel: [<ffffffff8005d28d>] tracesys+0xd5/0xe0 Sep 15 04:31:42 smtp2 kernel: [<ffffffff8005d427>] ptregscall_common+0x67/0xac Sep 15 04:31:42 smtp2 kernel: Sep 15 04:31:42 smtp2 kernel: Mem-info: Sep 15 04:31:42 smtp2 kernel: Node 0 DMA per-cpu: Sep 15 04:31:42 smtp2 kernel: cpu 0 hot: high 0, batch 1 used:0 Sep 15 04:31:42 smtp2 kernel: cpu 0 cold: high 0, batch 1 used:0 Sep 15 04:31:42 smtp2 kernel: cpu 1 hot: high 0, batch 1 used:0 Sep 15 04:31:42 smtp2 kernel: cpu 1 cold: high 0, batch 1 used:0 Sep 15 04:31:42 smtp2 kernel: Node 0 DMA32 per-cpu: Sep 15 04:31:42 smtp2 kernel: cpu 0 hot: high 186, batch 31 used:48 Sep 15 04:31:42 smtp2 kernel: cpu 0 cold: high 62, batch 15 used:40 Sep 15 04:31:42 smtp2 kernel: cpu 1 hot: high 186, batch 31 used:156 Sep 15 04:31:42 smtp2 kernel: cpu 1 cold: high 62, batch 15 used:60 Sep 15 04:31:42 smtp2 kernel: Node 0 Normal per-cpu: empty Sep 15 04:31:42 smtp2 kernel: Node 0 HighMem per-cpu: empty Sep 15 04:31:42 smtp2 kernel: Free pages: 7860kB (0kB HighMem) Sep 15 04:31:42 smtp2 kernel: Active:222971 inactive:131062 dirty:0 writeback:0 unstable:0 free:1965 slab:32477 mapped-file:2705 mapped-anon:353176 pagetables:97307 Sep 15 04:31:42 smtp2 kernel: Node 0 DMA free:2152kB min:28kB low:32kB high:40kB active:0kB inactive:0kB present:10716kB pages_scanned:0 all_unreclaimable? yes Sep 15 04:31:42 smtp2 kernel: lowmem_reserve[]: 0 2003 2003 2003 Sep 15 04:31:42 smtp2 kernel: Node 0 DMA32 free:5708kB min:5708kB low:7132kB high:8560kB active:891884kB inactive:524248kB present:2051184kB pages_scanned:2988074 all_unreclaimable? yes Sep 15 04:31:42 smtp2 kernel: lowmem_reserve[]: 0 0 0 0 Sep 15 04:31:42 smtp2 kernel: Node 0 Normal free:0kB min:0kB low:0kB high:0kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no Sep 15 04:31:42 smtp2 kernel: lowmem_reserve[]: 0 0 0 0 Sep 15 04:31:42 smtp2 kernel: Node 0 HighMem free:0kB min:128kB low:128kB high:128kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no Sep 15 04:31:42 smtp2 kernel: lowmem_reserve[]: 0 0 0 0 Sep 15 04:31:42 smtp2 kernel: Node 0 DMA: 2*4kB 4*8kB 2*16kB 5*32kB 4*64kB 1*128kB 2*256kB 0*512kB 1*1024kB 0*2048kB 0*4096kB = 2152kB Sep 15 04:31:42 smtp2 kernel: Node 0 DMA32: 129*4kB 1*8kB 2*16kB 13*32kB 4*64kB 1*128kB 1*256kB 0*512kB 0*1024kB 0*2048kB 1*4096kB = 5708kB Sep 15 04:31:42 smtp2 kernel: Node 0 Normal: empty Sep 15 04:31:42 smtp2 kernel: Node 0 HighMem: empty Sep 15 04:31:42 smtp2 kernel: 6436 pagecache pages Sep 15 04:31:42 smtp2 kernel: Swap cache: add 613470, delete 609770, find 99957/110522, race 3+8 Sep 15 04:31:42 smtp2 kernel: Free swap = 0kB Sep 15 04:31:42 smtp2 kernel: Total swap = 2096472kB Sep 15 04:31:42 smtp2 kernel: Free swap: 0kB Sep 15 04:31:42 smtp2 kernel: 524000 pages of RAM Sep 15 04:31:42 smtp2 kernel: 9347 reserved pages Sep 15 04:31:42 smtp2 kernel: 1057738 pages shared Sep 15 04:31:42 smtp2 kernel: 3700 pages swap cached Sep 15 04:31:42 smtp2 kernel: Out of memory: Killed process 3250 (crond). Sep 15 04:31:42 smtp2 kernel: postdrop invoked oom-killer: gfp_mask=0x201d2, order=0, oomkilladj=0 Sep 15 04:31:42 smtp2 kernel: Sep 15 04:31:42 smtp2 kernel: Call Trace: Sep 15 04:31:42 smtp2 kernel: [<ffffffff800c3bda>] out_of_memory+0x8e/0x2f3 Sep 15 04:31:42 smtp2 kernel: [<ffffffff8000f2ea>] __alloc_pages+0x245/0x2ce Sep 15 04:31:42 smtp2 kernel: [<ffffffff80012a77>] __do_page_cache_readahead+0xa2/0x1e6 Sep 15 04:31:42 smtp2 kernel: [<ffffffff80054ab8>] sock_sendmsg+0xf3/0x110 Sep 15 04:31:42 smtp2 kernel: [<ffffffff8000e87a>] link_path_walk+0xd3/0xe5 Sep 15 04:31:42 smtp2 kernel: [<ffffffff80013406>] filemap_nopage+0x148/0x322 Sep 15 04:31:42 smtp2 kernel: [<ffffffff8000886a>] __handle_mm_fault+0x1f8/0xe5c Sep 15 04:31:42 smtp2 kernel: [<ffffffff80066b9f>] do_page_fault+0x4cb/0x830 Sep 15 04:31:42 smtp2 kernel: [<ffffffff80059fdf>] hrtimer_cancel+0xc/0x16 Sep 15 04:31:42 smtp2 kernel: [<ffffffff80063db7>] do_nanosleep+0x47/0x70 Sep 15 04:31:42 smtp2 kernel: [<ffffffff80059ecc>] hrtimer_nanosleep+0x58/0x118 Sep 15 04:31:42 smtp2 kernel: [<ffffffff8005dde9>] error_exit+0x0/0x84 Sep 15 04:31:42 smtp2 kernel: Sep 15 04:31:42 smtp2 kernel: Mem-info: Sep 15 04:31:42 smtp2 kernel: Node 0 DMA per-cpu: Sep 15 04:31:42 smtp2 kernel: cpu 0 hot: high 0, batch 1 used:0 Sep 15 04:31:42 smtp2 kernel: cpu 0 cold: high 0, batch 1 used:0 Sep 15 04:31:42 smtp2 kernel: cpu 1 hot: high 0, batch 1 used:0 Sep 15 04:31:42 smtp2 kernel: cpu 1 cold: high 0, batch 1 used:0 Sep 15 04:31:42 smtp2 kernel: Node 0 DMA32 per-cpu: Sep 15 04:31:42 smtp2 kernel: cpu 0 hot: high 186, batch 31 used:57 Sep 15 04:31:42 smtp2 kernel: cpu 0 cold: high 62, batch 15 used:37 Sep 15 04:31:42 smtp2 kernel: cpu 1 hot: high 186, batch 31 used:166 Sep 15 04:31:42 smtp2 kernel: cpu 1 cold: high 62, batch 15 used:14 Sep 15 04:31:42 smtp2 kernel: Node 0 Normal per-cpu: empty Sep 15 04:31:42 smtp2 kernel: Node 0 HighMem per-cpu: empty Sep 15 04:31:42 smtp2 kernel: Free pages: 7984kB (0kB HighMem) Sep 15 04:31:42 smtp2 kernel: Active:182184 inactive:162368 dirty:0 writeback:0 unstable:0 free:1996 slab:32439 mapped-file:2705 mapped-anon:353176 pagetables:97307 Sep 15 04:31:42 smtp2 kernel: Node 0 DMA free:2152kB min:28kB low:32kB high:40kB active:0kB inactive:0kB present:10716kB pages_scanned:0 all_unreclaimable? yes Sep 15 04:31:42 smtp2 kernel: lowmem_reserve[]: 0 2003 2003 2003 Sep 15 04:31:42 smtp2 kernel: Node 0 DMA32 free:5832kB min:5708kB low:7132kB high:8560kB active:728576kB inactive:649760kB present:2051184kB pages_scanned:422478 all_unreclaimable? no Sep 15 04:31:42 smtp2 kernel: lowmem_reserve[]: 0 0 0 0 Sep 15 04:31:42 smtp2 kernel: Node 0 Normal free:0kB min:0kB low:0kB high:0kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no Sep 15 04:31:42 smtp2 kernel: lowmem_reserve[]: 0 0 0 0 Sep 15 04:31:42 smtp2 kernel: Node 0 HighMem free:0kB min:128kB low:128kB high:128kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no Sep 15 04:31:42 smtp2 kernel: lowmem_reserve[]: 0 0 0 0 Sep 15 04:31:42 smtp2 kernel: Node 0 DMA: 2*4kB 4*8kB 2*16kB 5*32kB 4*64kB 1*128kB 2*256kB 0*512kB 1*1024kB 0*2048kB 0*4096kB = 2152kB Sep 15 04:31:42 smtp2 kernel: Node 0 DMA32: 130*4kB 4*8kB 2*16kB 14*32kB 5*64kB 1*128kB 1*256kB 0*512kB 0*1024kB 0*2048kB 1*4096kB = 5832kB Sep 15 04:31:43 smtp2 kernel: Node 0 Normal: empty Sep 15 04:31:43 smtp2 kernel: Node 0 HighMem: empty Sep 15 04:31:43 smtp2 kernel: 6467 pagecache pages Sep 15 04:31:43 smtp2 kernel: Swap cache: add 613470, delete 609770, find 99957/110522, race 3+8 Sep 15 04:31:43 smtp2 kernel: Free swap = 0kB Sep 15 04:31:43 smtp2 kernel: Total swap = 2096472kB Sep 15 04:31:43 smtp2 kernel: Free swap: 0kB Sep 15 04:31:43 smtp2 kernel: 524000 pages of RAM Sep 15 04:31:43 smtp2 kernel: 9347 reserved pages Sep 15 04:31:43 smtp2 kernel: 1067643 pages shared Sep 15 04:31:43 smtp2 kernel: 3700 pages swap cached Sep 15 04:31:43 smtp2 kernel: postdrop invoked oom-killer: gfp_mask=0x201d2, order=0, oomkilladj=0 Sep 15 04:31:43 smtp2 kernel: Sep 15 04:31:43 smtp2 kernel: Call Trace: Sep 15 04:31:43 smtp2 kernel: [<ffffffff800c3bda>] out_of_memory+0x8e/0x2f3 Sep 15 04:31:43 smtp2 kernel: [<ffffffff8000f2ea>] __alloc_pages+0x245/0x2ce Sep 15 04:31:43 smtp2 kernel: [<ffffffff80012a77>] __do_page_cache_readahead+0xa2/0x1e6 Sep 15 04:31:43 smtp2 kernel: [<ffffffff80054ab8>] sock_sendmsg+0xf3/0x110 Sep 15 04:31:43 smtp2 kernel: [<ffffffff8000e87a>] link_path_walk+0xd3/0xe5 Sep 15 04:31:43 smtp2 kernel: [<ffffffff80013406>] filemap_nopage+0x148/0x322 Sep 15 04:31:43 smtp2 kernel: [<ffffffff8000886a>] __handle_mm_fault+0x1f8/0xe5c Sep 15 04:31:43 smtp2 kernel: [<ffffffff80066b9f>] do_page_fault+0x4cb/0x830 Sep 15 04:31:43 smtp2 kernel: [<ffffffff80059fdf>] hrtimer_cancel+0xc/0x16 Sep 15 04:31:43 smtp2 kernel: [<ffffffff80063db7>] do_nanosleep+0x47/0x70 Sep 15 04:31:43 smtp2 kernel: [<ffffffff80059ecc>] hrtimer_nanosleep+0x58/0x118 Sep 15 04:31:43 smtp2 kernel: [<ffffffff8005dde9>] error_exit+0x0/0x84 Sep 15 04:31:43 smtp2 kernel: Sep 15 04:31:43 smtp2 kernel: Mem-info: Sep 15 04:31:43 smtp2 kernel: Node 0 DMA per-cpu: Sep 15 04:31:43 smtp2 kernel: cpu 0 hot: high 0, batch 1 used:0 Sep 15 04:31:43 smtp2 kernel: cpu 0 cold: high 0, batch 1 used:0 Sep 15 04:31:43 smtp2 kernel: cpu 1 hot: high 0, batch 1 used:0 Sep 15 04:31:43 smtp2 kernel: cpu 1 cold: high 0, batch 1 used:0 Sep 15 04:31:43 smtp2 kernel: Node 0 DMA32 per-cpu: Sep 15 04:31:43 smtp2 kernel: cpu 0 hot: high 186, batch 31 used:113 Sep 15 04:31:43 smtp2 kernel: cpu 0 cold: high 62, batch 15 used:32 Sep 15 04:31:43 smtp2 kernel: cpu 1 hot: high 186, batch 31 used:171 Sep 15 04:31:43 smtp2 kernel: cpu 1 cold: high 62, batch 15 used:57 Sep 15 04:31:43 smtp2 kernel: Node 0 Normal per-cpu: empty Sep 15 04:31:43 smtp2 kernel: Node 0 HighMem per-cpu: empty Sep 15 04:31:43 smtp2 kernel: Free pages: 8388kB (0kB HighMem) Sep 15 04:31:43 smtp2 kernel: Active:180176 inactive:164113 dirty:0 writeback:0 unstable:0 free:2097 slab:32458 mapped-file:2711 mapped-anon:352991 pagetables:97276 Sep 15 04:31:43 smtp2 kernel: Node 0 DMA free:2152kB min:28kB low:32kB high:40kB active:0kB inactive:0kB present:10716kB pages_scanned:0 all_unreclaimable? yes Sep 15 04:31:43 smtp2 kernel: lowmem_reserve[]: 0 2003 2003 2003 Sep 15 04:31:43 smtp2 kernel: Node 0 DMA32 free:6236kB min:5708kB low:7132kB high:8560kB active:719796kB inactive:657360kB present:2051184kB pages_scanned:531325 all_unreclaimable? no Sep 15 04:31:43 smtp2 kernel: lowmem_reserve[]: 0 0 0 0 Sep 15 04:31:43 smtp2 kernel: Node 0 Normal free:0kB min:0kB low:0kB high:0kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no Sep 15 04:31:43 smtp2 kernel: lowmem_reserve[]: 0 0 0 0 Sep 15 04:31:43 smtp2 kernel: Node 0 HighMem free:0kB min:128kB low:128kB high:128kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no Sep 15 04:31:43 smtp2 kernel: lowmem_reserve[]: 0 0 0 0 Sep 15 04:31:43 smtp2 kernel: Node 0 DMA: 2*4kB 4*8kB 2*16kB 5*32kB 4*64kB 1*128kB 2*256kB 0*512kB 1*1024kB 0*2048kB 0*4096kB = 2152kB Sep 15 04:31:43 smtp2 kernel: Node 0 DMA32: 235*4kB 2*8kB 2*16kB 14*32kB 5*64kB 1*128kB 1*256kB 0*512kB 0*1024kB 0*2048kB 1*4096kB = 6236kB Sep 15 04:31:43 smtp2 kernel: Node 0 Normal: empty Sep 15 04:31:43 smtp2 kernel: Node 0 HighMem: empty Sep 15 04:31:43 smtp2 kernel: 6501 pagecache pages Sep 15 04:31:43 smtp2 kernel: Swap cache: add 613655, delete 609955, find 99957/110524, race 3+8 Sep 15 04:31:44 smtp2 kernel: Free swap = 0kB Sep 15 04:31:44 smtp2 kernel: Total swap = 2096472kB Sep 15 04:31:45 smtp2 kernel: Free swap: 0kB Sep 15 04:31:45 smtp2 kernel: 524000 pages of RAM Sep 15 04:31:45 smtp2 kernel: 9347 reserved pages Sep 15 04:31:46 smtp2 kernel: 1066858 pages shared Sep 15 04:31:46 smtp2 kernel: 3700 pages swap cached Sep 15 04:31:46 smtp2 kernel: Out of memory: Killed process 3268 (crond). Sep 15 04:31:46 smtp2 kernel: block drbd0: helper command: /sbin/drbdadm after-resync-target minor-0 exit code 0 (0x0) Sep 15 04:31:46 smtp2 kernel: block drbd0: md_sync_timer expired! Worker calls drbd_md_sync(). Sep 15 04:31:46 smtp2 kernel: modclusterd invoked oom-killer: gfp_mask=0x201d2, order=0, oomkilladj=0 Sep 15 04:31:46 smtp2 kernel: Sep 15 04:31:46 smtp2 kernel: Call Trace: Sep 15 04:31:46 smtp2 kernel: [<ffffffff800c3bda>] out_of_memory+0x8e/0x2f3 Sep 15 04:31:46 smtp2 kernel: [<ffffffff8000f2ea>] __alloc_pages+0x245/0x2ce Sep 15 04:31:46 smtp2 kernel: [<ffffffff80012a77>] __do_page_cache_readahead+0xa2/0x1e6 Sep 15 04:31:46 smtp2 kernel: [<ffffffff80063a05>] __wait_on_bit_lock+0x5b/0x66 Sep 15 04:31:46 smtp2 kernel: [<ffffffff8003f626>] __lock_page+0x5e/0x64 Sep 15 04:31:46 smtp2 kernel: [<ffffffff80013406>] filemap_nopage+0x148/0x322 Sep 15 04:31:46 smtp2 kernel: [<ffffffff8000886a>] __handle_mm_fault+0x1f8/0xe5c Sep 15 04:31:46 smtp2 kernel: [<ffffffff80066b9f>] do_page_fault+0x4cb/0x830 Sep 15 04:31:46 smtp2 kernel: [<ffffffff800320b8>] sys_faccessat+0xe4/0x18d Sep 15 04:31:46 smtp2 kernel: [<ffffffff800b4629>] audit_syscall_entry+0x14a/0x1b3 Sep 15 04:31:46 smtp2 kernel: [<ffffffff8005dde9>] error_exit+0x0/0x84 Sep 15 04:31:46 smtp2 kernel: Sep 15 04:31:46 smtp2 kernel: Mem-info: Sep 15 04:31:47 smtp2 kernel: Node 0 DMA per-cpu: Sep 15 04:31:47 smtp2 kernel: cpu 0 hot: high 0, batch 1 used:0 Sep 15 04:31:47 smtp2 kernel: cpu 0 cold: high 0, batch 1 used:0 Sep 15 04:31:47 smtp2 kernel: cpu 1 hot: high 0, batch 1 used:0 Sep 15 04:31:47 smtp2 kernel: cpu 1 cold: high 0, batch 1 used:0 Sep 15 04:31:47 smtp2 kernel: Node 0 DMA32 per-cpu: Sep 15 04:31:47 smtp2 kernel: cpu 0 hot: high 186, batch 31 used:30 Sep 15 04:31:47 smtp2 kernel: cpu 0 cold: high 62, batch 15 used:14 Sep 15 04:31:47 smtp2 kernel: cpu 1 hot: high 186, batch 31 used:6 Sep 15 04:31:47 smtp2 kernel: cpu 1 cold: high 62, batch 15 used:31 Sep 15 04:31:47 smtp2 kernel: Node 0 Normal per-cpu: empty Sep 15 04:31:47 smtp2 kernel: Node 0 HighMem per-cpu: empty Sep 15 04:31:47 smtp2 kernel: Free pages: 7804kB (0kB HighMem) Sep 15 04:31:47 smtp2 kernel: Active:223476 inactive:131748 dirty:0 writeback:0 unstable:0 free:1951 slab:32653 mapped-file:2757 mapped-anon:353073 pagetables:97208 Sep 15 04:31:47 smtp2 kernel: Node 0 DMA free:2152kB min:28kB low:32kB high:40kB active:0kB inactive:0kB present:10716kB pages_scanned:0 all_unreclaimable? yes Sep 15 04:31:47 smtp2 kernel: lowmem_reserve[]: 0 2003 2003 2003 Sep 15 04:31:47 smtp2 kernel: Node 0 DMA32 free:5652kB min:5708kB low:7132kB high:8560kB active:895736kB inactive:525032kB present:2051184kB pages_scanned:3253851 all_unreclaimable? yes Sep 15 04:31:47 smtp2 kernel: lowmem_reserve[]: 0 0 0 0 Sep 15 04:31:47 smtp2 kernel: Node 0 Normal free:0kB min:0kB low:0kB high:0kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no Sep 15 04:31:47 smtp2 kernel: lowmem_reserve[]: 0 0 0 0 Sep 15 04:31:47 smtp2 kernel: Node 0 HighMem free:0kB min:128kB low:128kB high:128kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no Sep 15 04:31:47 smtp2 kernel: lowmem_reserve[]: 0 0 0 0 Sep 15 04:31:47 smtp2 kernel: Node 0 DMA: 2*4kB 4*8kB 2*16kB 5*32kB 4*64kB 1*128kB 2*256kB 0*512kB 1*1024kB 0*2048kB 0*4096kB = 2152kB Sep 15 04:31:47 smtp2 kernel: Node 0 DMA32: 19*4kB 27*8kB 3*16kB 12*32kB 5*64kB 2*128kB 1*256kB 0*512kB 0*1024kB 0*2048kB 1*4096kB = 5652kB Sep 15 04:31:47 smtp2 kernel: Node 0 Normal: empty Sep 15 04:31:47 smtp2 kernel: Node 0 HighMem: empty Sep 15 04:31:47 smtp2 kernel: 6741 pagecache pages Sep 15 04:31:47 smtp2 kernel: Swap cache: add 616939, delete 613225, find 100005/110937, race 3+9 Sep 15 04:31:47 smtp2 kernel: Free swap = 0kB Sep 15 04:31:47 smtp2 kernel: Total swap = 2096472kB Sep 15 04:31:47 smtp2 kernel: Free swap: 0kB Sep 15 04:31:47 smtp2 kernel: 524000 pages of RAM Sep 15 04:31:47 smtp2 kernel: 9347 reserved pages Sep 15 04:31:47 smtp2 kernel: 1057504 pages shared Sep 15 04:31:47 smtp2 kernel: 3714 pages swap cached Sep 15 04:31:47 smtp2 kernel: Out of memory: Killed process 3284 (crond). Sep 15 04:31:47 smtp2 kernel: modclusterd invoked oom-killer: gfp_mask=0x201d2, order=0, oomkilladj=0 Sep 15 04:31:47 smtp2 kernel: Sep 15 04:31:47 smtp2 kernel: Call Trace: Sep 15 04:31:47 smtp2 kernel: [<ffffffff800c3bda>] out_of_memory+0x8e/0x2f3 Sep 15 04:31:47 smtp2 kernel: [<ffffffff8000f2ea>] __alloc_pages+0x245/0x2ce Sep 15 04:31:47 smtp2 kernel: [<ffffffff80012a77>] __do_page_cache_readahead+0xa2/0x1e6 Sep 15 04:31:47 smtp2 kernel: [<ffffffff80063a05>] __wait_on_bit_lock+0x5b/0x66 Sep 15 04:31:47 smtp2 kernel: [<ffffffff8003f626>] __lock_page+0x5e/0x64 Sep 15 04:31:47 smtp2 kernel: [<ffffffff80013406>] filemap_nopage+0x148/0x322 Sep 15 04:31:52 smtp2 kernel: [<ffffffff8000886a>] __handle_mm_fault+0x1f8/0xe5c Sep 15 04:31:52 smtp2 kernel: [<ffffffff80066b9f>] do_page_fault+0x4cb/0x830 Sep 15 04:31:52 smtp2 kernel: [<ffffffff800320b8>] sys_faccessat+0xe4/0x18d Sep 15 04:31:52 smtp2 kernel: [<ffffffff800b4629>] audit_syscall_entry+0x14a/0x1b3 Sep 15 04:31:52 smtp2 kernel: [<ffffffff8005dde9>] error_exit+0x0/0x84 Sep 15 04:31:52 smtp2 kernel: Sep 15 04:31:52 smtp2 kernel: Mem-info: Sep 15 04:31:52 smtp2 kernel: Node 0 DMA per-cpu: Sep 15 04:31:52 smtp2 kernel: cpu 0 hot: high 0, batch 1 used:0 Sep 15 04:31:52 smtp2 kernel: cpu 0 cold: high 0, batch 1 used:0 Sep 15 04:31:52 smtp2 kernel: cpu 1 hot: high 0, batch 1 used:0 Sep 15 04:31:52 smtp2 kernel: cpu 1 cold: high 0, batch 1 used:0 Sep 15 04:31:52 smtp2 kernel: Node 0 DMA32 per-cpu: Sep 15 04:31:54 smtp2 kernel: cpu 0 hot: high 186, batch 31 used:0 Sep 15 04:31:54 smtp2 kernel: cpu 0 cold: high 62, batch 15 used:14 Sep 15 04:31:54 smtp2 kernel: cpu 1 hot: high 186, batch 31 used:58 Sep 15 04:31:54 smtp2 kernel: cpu 1 cold: high 62, batch 15 used:26 Sep 15 04:31:54 smtp2 kernel: Node 0 Normal per-cpu: empty Sep 15 04:31:54 smtp2 kernel: Node 0 HighMem per-cpu: empty Sep 15 04:31:54 smtp2 kernel: Free pages: 7812kB (0kB HighMem) Sep 15 04:31:54 smtp2 kernel: Active:114250 inactive:241428 dirty:0 writeback:1 unstable:0 free:1953 slab:32655 mapped-file:2751 mapped-anon:353072 pagetables:97208 Sep 15 04:31:54 smtp2 kernel: Node 0 DMA free:2152kB min:28kB low:32kB high:40kB active:0kB inactive:0kB present:10716kB pages_scanned:0 all_unreclaimable? yes Sep 15 04:31:54 smtp2 kernel: lowmem_reserve[]: 0 2003 2003 2003 Sep 15 04:31:54 smtp2 kernel: Node 0 DMA32 free:5660kB min:5708kB low:7132kB high:8560kB active:457000kB inactive:965712kB present:2051184kB pages_scanned:4264628 all_unreclaimable? yes Sep 15 04:31:54 smtp2 kernel: lowmem_reserve[]: 0 0 0 0 Sep 15 04:31:55 smtp2 kernel: Node 0 Normal free:0kB min:0kB low:0kB high:0kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no Sep 15 04:31:55 smtp2 kernel: lowmem_reserve[]: 0 0 0 0 Sep 15 04:31:55 smtp2 kernel: Node 0 HighMem free:0kB min:128kB low:128kB high:128kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no Sep 15 04:31:55 smtp2 kernel: lowmem_reserve[]: 0 0 0 0 Sep 15 04:31:55 smtp2 kernel: Node 0 DMA: 2*4kB 4*8kB 2*16kB 5*32kB 4*64kB 1*128kB 2*256kB 0*512kB 1*1024kB 0*2048kB 0*4096kB = 2152kB Sep 15 04:31:55 smtp2 kernel: Node 0 DMA32: 17*4kB 29*8kB 3*16kB 12*32kB 5*64kB 2*128kB 1*256kB 0*512kB 0*1024kB 0*2048kB 1*4096kB = 5660kB Sep 15 04:31:55 smtp2 kernel: Node 0 Normal: empty Sep 15 04:31:55 smtp2 kernel: Node 0 HighMem: empty Sep 15 04:31:55 smtp2 kernel: 6705 pagecache pages Sep 15 04:31:55 smtp2 kernel: Swap cache: add 616966, delete 613253, find 100005/110937, race 3+9 Sep 15 04:31:55 smtp2 kernel: Free swap = 0kB Sep 15 04:31:55 smtp2 kernel: Total swap = 2096472kB Sep 15 04:31:55 smtp2 kernel: Free swap: 0kB Sep 15 04:31:55 smtp2 kernel: 524000 pages of RAM Sep 15 04:31:55 smtp2 kernel: 9347 reserved pages Sep 15 04:31:55 smtp2 kernel: 1055243 pages shared Sep 15 04:31:55 smtp2 kernel: 3713 pages swap cached Sep 15 04:31:59 smtp2 nss-mysql[24311]: _nss_mysql_db_connect: connection failed: Lost connection to MySQL server at 'reading authorization packet', system error: 0 Sep 15 04:31:59 smtp2 nss-mysql[24311]: setent(1): connection to the database failed. Sep 15 04:35:20 smtp2 kernel: modclusterd invoked oom-killer: gfp_mask=0x201d2, order=0, oomkilladj=0 Sep 15 04:35:27 smtp2 kernel: Sep 15 04:35:27 smtp2 kernel: Call Trace: Sep 15 04:35:29 smtp2 kernel: [<ffffffff800c3bda>] out_of_memory+0x8e/0x2f3 Sep 15 04:35:29 smtp2 kernel: [<ffffffff8000f2ea>] __alloc_pages+0x245/0x2ce Sep 15 04:35:29 smtp2 kernel: [<ffffffff80012a77>] __do_page_cache_readahead+0xa2/0x1e6 Sep 15 04:35:29 smtp2 kernel: [<ffffffff80063a05>] __wait_on_bit_lock+0x5b/0x66 Sep 15 04:35:29 smtp2 kernel: [<ffffffff8003f626>] __lock_page+0x5e/0x64 Sep 15 04:35:29 smtp2 kernel: [<ffffffff80013406>] filemap_nopage+0x148/0x322 Sep 15 04:35:29 smtp2 kernel: [<ffffffff8000886a>] __handle_mm_fault+0x1f8/0xe5c Sep 15 04:35:29 smtp2 kernel: [<ffffffff80066b9f>] do_page_fault+0x4cb/0x830 Sep 15 04:35:29 smtp2 kernel: [<ffffffff800320b8>] sys_faccessat+0xe4/0x18d Sep 15 04:35:29 smtp2 kernel: [<ffffffff800b4629>] audit_syscall_entry+0x14a/0x1b3 Sep 15 04:35:29 smtp2 kernel: [<ffffffff8005dde9>] error_exit+0x0/0x84 Sep 15 04:35:29 smtp2 kernel: Sep 15 04:35:30 smtp2 kernel: Mem-info: Sep 15 04:35:31 smtp2 snmpd[2489]: Connection from UDP: [206.16.46.12]:59774 Sep 15 04:35:31 smtp2 kernel: Node 0 DMA per-cpu: Sep 15 04:35:31 smtp2 snmpd[2489]: Received SNMP packet(s) from UDP: [206.16.46.12]:59774 Sep 15 04:35:31 smtp2 kernel: cpu 0 hot: high 0, batch 1 used:0 Sep 15 04:35:32 smtp2 snmpd[2489]: Connection from UDP: [206.16.46.12]:59774 Sep 15 04:35:32 smtp2 kernel: cpu 0 cold: high 0, batch 1 used:0 Sep 15 04:35:32 smtp2 snmpd[2489]: Connection from UDP: [206.16.46.12]:59774 Sep 15 04:35:32 smtp2 kernel: cpu 1 hot: high 0, batch 1 used:0 Sep 15 04:35:32 smtp2 snmpd[2489]: Connection from UDP: [206.16.46.12]:59774 Sep 15 04:35:32 smtp2 kernel: cpu 1 cold: high 0, batch 1 used:0 Sep 15 04:35:32 smtp2 snmpd[2489]: Connection from UDP: [206.16.46.12]:59774 Sep 15 04:35:32 smtp2 kernel: Node 0 DMA32 per-cpu: Sep 15 04:35:32 smtp2 snmpd[2489]: Connection from UDP: [206.16.46.12]:59774 Sep 15 04:35:32 smtp2 kernel: cpu 0 hot: high 186, batch 31 used:30 Sep 15 04:35:32 smtp2 snmpd[2489]: Connection from UDP: [206.16.46.12]:59774 Sep 15 04:35:32 smtp2 kernel: cpu 0 cold: high 62, batch 15 used:14 Sep 15 04:35:32 smtp2 snmpd[2489]: Connection from UDP: [206.16.46.12]:59774 Sep 15 04:35:32 smtp2 kernel: cpu 1 hot: high 186, batch 31 used:44 Sep 15 04:35:32 smtp2 snmpd[2489]: Connection from UDP: [206.16.46.12]:59774 Sep 15 04:35:32 smtp2 kernel: cpu 1 cold: high 62, batch 15 used:58 Sep 15 04:35:32 smtp2 snmpd[2489]: Connection from UDP: [206.16.46.12]:59774 Sep 15 04:35:32 smtp2 kernel: Node 0 Normal per-cpu: empty Sep 15 04:35:32 smtp2 snmpd[2489]: Connection from UDP: [206.16.46.12]:59774 Sep 15 04:35:32 smtp2 kernel: Node 0 HighMem per-cpu: empty Sep 15 04:35:32 smtp2 snmpd[2489]: Connection from UDP: [206.16.46.12]:59774 Sep 15 04:35:32 smtp2 kernel: Free pages: 7824kB (0kB HighMem) Sep 15 04:35:32 smtp2 snmpd[2489]: Connection from UDP: [206.16.46.12]:59774 Sep 15 04:35:32 smtp2 kernel: Active:174258 inactive:180606 dirty:0 writeback:0 unstable:0 free:1956 slab:32676 mapped-file:2723 mapped-anon:353207 pagetables:97216 Sep 15 04:35:32 smtp2 snmpd[2489]: Connection from UDP: [206.16.46.12]:59774 Sep 15 04:35:32 smtp2 kernel: Node 0 DMA free:2152kB min:28kB low:32kB high:40kB active:0kB inactive:0kB present:10716kB pages_scanned:0 all_unreclaimable? yes Sep 15 04:35:32 smtp2 snmpd[2489]: Connection from UDP: [206.16.46.12]:59774 Sep 15 04:35:32 smtp2 kernel: lowmem_reserve[]: 0 2003 2003 2003 Sep 15 04:35:32 smtp2 snmpd[2489]: Connection from UDP: [206.16.46.12]:59774 Sep 15 04:35:33 smtp2 kernel: Node 0 DMA32 free:5672kB min:5708kB low:7132kB high:8560kB active:697032kB inactive:722424kB present:2051184kB pages_scanned:2156508 all_unreclaimable? yes Sep 15 04:35:33 smtp2 snmpd[2489]: Connection from UDP: [206.16.46.12]:59774 Sep 15 04:35:33 smtp2 kernel: lowmem_reserve[]: 0 0 0 0 Sep 15 04:35:33 smtp2 snmpd[2489]: Connection from UDP: [206.16.46.12]:59774 Sep 15 04:35:33 smtp2 kernel: Node 0 Normal free:0kB min:0kB low:0kB high:0kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no Sep 15 04:35:33 smtp2 snmpd[2489]: Connection from UDP: [206.16.46.12]:59774 Sep 15 04:35:33 smtp2 kernel: lowmem_reserve[]: 0 0 0 0 Sep 15 04:35:33 smtp2 snmpd[2489]: Connection from UDP: [206.16.46.12]:59774 Sep 15 04:35:33 smtp2 kernel: Node 0 HighMem free:0kB min:128kB low:128kB high:128kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no Sep 15 04:35:33 smtp2 snmpd[2489]: Connection from UDP: [206.16.46.12]:59774 Sep 15 04:35:33 smtp2 kernel: lowmem_reserve[]: 0 0 0 0 Sep 15 04:35:33 smtp2 snmpd[2489]: Connection from UDP: [206.16.46.12]:59774 Sep 15 04:35:33 smtp2 kernel: Node 0 DMA: 2*4kB 4*8kB 2*16kB 5*32kB 4*64kB 1*128kB 2*256kB 0*512kB 1*1024kB 0*2048kB 0*4096kB = 2152kB Sep 15 04:35:33 smtp2 snmpd[2489]: Connection from UDP: [206.16.46.12]:59774 Sep 15 04:35:33 smtp2 kernel: Node 0 DMA32: 12*4kB 33*8kB 3*16kB 12*32kB 5*64kB 2*128kB 1*256kB 0*512kB 0*1024kB 0*2048kB 1*4096kB = 5672kB Sep 15 04:35:33 smtp2 snmpd[2489]: Connection from UDP: [206.16.46.12]:59774 Sep 15 04:35:33 smtp2 kernel: Node 0 Normal: empty Sep 15 04:35:33 smtp2 snmpd[2489]: Connection from UDP: [206.16.46.12]:59774 Sep 15 04:35:33 smtp2 kernel: Node 0 HighMem: empty Sep 15 04:35:33 smtp2 snmpd[2489]: Connection from UDP: [206.16.46.12]:59774 Sep 15 04:35:33 smtp2 kernel: 6490 pagecache pages Sep 15 04:35:33 smtp2 snmpd[2489]: Connection from UDP: [206.16.46.12]:59774 Sep 15 04:35:33 smtp2 kernel: Swap cache: add 618956, delete 615255, find 100105/111249, race 3+10 Sep 15 04:35:33 smtp2 snmpd[2489]: Connection from UDP: [206.16.46.12]:59774 Sep 15 04:35:33 smtp2 kernel: Free swap = 0kB Sep 15 04:35:33 smtp2 snmpd[2489]: Connection from UDP: [206.16.46.12]:59774 Sep 15 04:35:33 smtp2 kernel: Total swap = 2096472kB Sep 15 04:35:33 smtp2 snmpd[2489]: Connection from UDP: [206.16.46.12]:59774 Sep 15 04:35:33 smtp2 kernel: Free swap: 0kB Sep 15 04:35:33 smtp2 snmpd[2489]: Connection from UDP: [206.16.46.12]:59774 Sep 15 04:35:33 smtp2 kernel: 524000 pages of RAM Sep 15 04:35:33 smtp2 snmpd[2489]: Connection from UDP: [206.16.46.12]:59774 Sep 15 04:35:33 smtp2 kernel: 9347 reserved pages Sep 15 04:35:33 smtp2 snmpd[2489]: Connection from UDP: [206.16.46.12]:59774 Sep 15 04:35:33 smtp2 kernel: 1055994 pages shared Sep 15 04:35:33 smtp2 snmpd[2489]: Connection from UDP: [206.16.46.12]:59774 Sep 15 04:35:33 smtp2 kernel: 3701 pages swap cached Sep 15 04:35:33 smtp2 snmpd[2489]: Connection from UDP: [206.16.46.12]:59774 Sep 15 04:35:33 smtp2 kernel: Out of memory: Killed process 3301 (crond). Sep 15 04:35:34 smtp2 snmpd[2489]: Connection from UDP: [206.16.46.12]:59774 Sep 15 04:35:35 smtp2 kernel: postdrop invoked oom-killer: gfp_mask=0x201d2, order=0, oomkilladj=0 Sep 15 04:35:36 smtp2 snmpd[2489]: Connection from UDP: [206.16.46.12]:59774 Sep 15 04:35:36 smtp2 kernel: Sep 15 04:36:01 smtp2 snmpd[2489]: Connection from UDP: [206.16.46.12]:59774 Sep 15 04:36:07 smtp2 kernel: Call Trace: Sep 15 04:36:15 smtp2 kernel: [<ffffffff800c3bda>] out_of_memory+0x8e/0x2f3 Sep 15 04:36:15 smtp2 kernel: [<ffffffff8000f2ea>] __alloc_pages+0x245/0x2ce Sep 15 04:36:15 smtp2 kernel: [<ffffffff80012a77>] __do_page_cache_readahead+0xa2/0x1e6 Sep 15 04:36:15 smtp2 kernel: [<ffffffff80063a05>] __wait_on_bit_lock+0x5b/0x66 Sep 15 04:36:15 smtp2 kernel: [<ffffffff8003f626>] __lock_page+0x5e/0x64 Sep 15 04:36:15 smtp2 kernel: [<ffffffff80013406>] filemap_nopage+0x148/0x322 Sep 15 04:36:15 smtp2 kernel: [<ffffffff8000886a>] __handle_mm_fault+0x1f8/0xe5c Sep 15 04:36:15 smtp2 kernel: [<ffffffff80066b9f>] do_page_fault+0x4cb/0x830 Sep 15 04:36:15 smtp2 kernel: [<ffffffff80059fdf>] hrtimer_cancel+0xc/0x16 Sep 15 04:36:15 smtp2 kernel: [<ffffffff80063db7>] do_nanosleep+0x47/0x70 Sep 15 04:36:15 smtp2 kernel: [<ffffffff80059ecc>] hrtimer_nanosleep+0x58/0x118 Sep 15 04:36:15 smtp2 kernel: [<ffffffff8005dde9>] error_exit+0x0/0x84 Sep 15 04:36:15 smtp2 kernel: Sep 15 04:36:16 smtp2 kernel: Mem-info: Sep 15 04:36:16 smtp2 kernel: Node 0 DMA per-cpu: Sep 15 04:36:16 smtp2 kernel: cpu 0 hot: high 0, batch 1 used:0 Sep 15 04:36:16 smtp2 kernel: cpu 0 cold: high 0, batch 1 used:0 Sep 15 04:36:16 smtp2 kernel: cpu 1 hot: high 0, batch 1 used:0 Sep 15 04:36:16 smtp2 kernel: cpu 1 cold: high 0, batch 1 used:0 Sep 15 04:36:16 smtp2 kernel: Node 0 DMA32 per-cpu: Sep 15 04:36:16 smtp2 kernel: cpu 0 hot: high 186, batch 31 used:30 Sep 15 04:36:16 smtp2 kernel: cpu 0 cold: high 62, batch 15 used:45 Sep 15 04:36:16 smtp2 kernel: cpu 1 hot: high 186, batch 31 used:48 Sep 15 04:36:16 smtp2 kernel: cpu 1 cold: high 62, batch 15 used:58 Sep 15 04:36:16 smtp2 kernel: Node 0 Normal per-cpu: empty Sep 15 04:36:16 smtp2 kernel: Node 0 HighMem per-cpu: empty Sep 15 04:36:16 smtp2 kernel: Free pages: 7824kB (0kB HighMem) Sep 15 04:36:16 smtp2 kernel: Active:223642 inactive:131077 dirty:0 writeback:0 unstable:0 free:1956 slab:32672 mapped-file:2722 mapped-anon:353207 pagetables:97216 Sep 15 04:36:16 smtp2 snmpd[2489]: Connection from UDP: [206.16.46.12]:59774 Sep 15 04:36:16 smtp2 kernel: Node 0 DMA free:2152kB min:28kB low:32kB high:40kB active:0kB inactive:0kB present:10716kB pages_scanned:0 all_unreclaimable? yes Sep 15 04:36:16 smtp2 snmpd[2489]: Connection from UDP: [206.16.46.12]:59774 Sep 15 04:36:16 smtp2 kernel: lowmem_reserve[]: 0 2003 2003 2003 Sep 15 04:36:16 smtp2 snmpd[2489]: Connection from UDP: [206.16.46.12]:59774 Sep 15 04:36:16 smtp2 kernel: Node 0 DMA32 free:5672kB min:5708kB low:7132kB high:8560kB active:894576kB inactive:524428kB present:2051184kB pages_scanned:5377722 all_unreclaimable? yes Sep 15 04:36:16 smtp2 snmpd[2489]: Connection from UDP: [206.16.46.12]:59774 Sep 15 04:36:16 smtp2 kernel: lowmem_reserve[]: 0 0 0 0 Sep 15 04:36:16 smtp2 snmpd[2489]: Connection from UDP: [206.16.46.12]:59774 Sep 15 04:36:16 smtp2 kernel: Node 0 Normal free:0kB min:0kB low:0kB high:0kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no Sep 15 04:36:16 smtp2 kernel: lowmem_reserve[]: 0 0 0 0 Sep 15 04:36:16 smtp2 kernel: Node 0 HighMem free:0kB min:128kB low:128kB high:128kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no Sep 15 04:36:16 smtp2 kernel: lowmem_reserve[]: 0 0 0 0 Sep 15 04:36:16 smtp2 snmpd[2489]: Connection from UDP: [206.16.46.12]:59774 Sep 15 04:36:16 smtp2 kernel: Node 0 DMA: 2*4kB 4*8kB 2*16kB 5*32kB 4*64kB 1*128kB 2*256kB 0*512kB 1*1024kB 0*2048kB 0*4096kB = 2152kB Sep 15 04:36:16 smtp2 snmpd[2489]: Connection from UDP: [206.16.46.12]:59774 Sep 15 04:36:16 smtp2 kernel: Node 0 DMA32: 12*4kB 33*8kB 3*16kB 12*32kB 5*64kB 2*128kB 1*256kB 0*512kB 0*1024kB 0*2048kB 1*4096kB = 5672kB Sep 15 04:36:16 smtp2 snmpd[2489]: Connection from UDP: [206.16.46.12]:59774 Sep 15 04:36:16 smtp2 kernel: Node 0 Normal: empty Sep 15 04:36:16 smtp2 snmpd[2489]: Connection from UDP: [206.16.46.12]:59774 Sep 15 04:36:16 smtp2 kernel: Node 0 HighMem: empty Sep 15 04:36:16 smtp2 snmpd[2489]: Connection from UDP: [206.16.46.12]:59774 Sep 15 04:36:16 smtp2 kernel: 6459 pagecache pages Sep 15 04:36:16 smtp2 snmpd[2489]: Connection from UDP: [206.16.46.12]:59774 Sep 15 04:36:16 smtp2 kernel: Swap cache: add 618956, delete 615255, find 100105/111249, race 3+10 Sep 15 04:36:16 smtp2 snmpd[2489]: Connection from UDP: [206.16.46.12]:59774 Sep 15 04:36:16 smtp2 kernel: Free swap = 0kB Sep 15 04:36:16 smtp2 snmpd[2489]: Connection from UDP: [206.16.46.12]:59774 Sep 15 04:36:16 smtp2 kernel: Total swap = 2096472kB Sep 15 04:36:16 smtp2 snmpd[2489]: Connection from UDP: [206.16.46.12]:59774 Sep 15 04:36:16 smtp2 kernel: Free swap: 0kB Sep 15 04:36:16 smtp2 snmpd[2489]: Connection from UDP: [206.16.46.12]:59774 Sep 15 04:36:16 smtp2 kernel: 524000 pages of RAM Sep 15 04:36:16 smtp2 snmpd[2489]: Connection from UDP: [206.16.46.12]:59774 Sep 15 04:36:16 smtp2 kernel: 9347 reserved pages Sep 15 04:36:16 smtp2 snmpd[2489]: Connection from UDP: [206.16.46.12]:59774 Sep 15 04:36:16 smtp2 kernel: 1056093 pages shared Sep 15 04:36:16 smtp2 snmpd[2489]: Connection from UDP: [206.16.46.12]:59774 Sep 15 04:36:16 smtp2 kernel: 3701 pages swap cached Sep 15 04:36:16 smtp2 snmpd[2489]: Connection from UDP: [206.16.46.12]:59774 Sep 15 04:36:16 smtp2 kernel: postdrop invoked oom-killer: gfp_mask=0x4d0, order=0, oomkilladj=0 Sep 15 04:36:16 smtp2 snmpd[2489]: Connection from UDP: [206.16.46.12]:59774 Sep 15 04:36:16 smtp2 kernel: Sep 15 04:36:16 smtp2 snmpd[2489]: Connection from UDP: [206.16.46.12]:59774 Sep 15 04:36:16 smtp2 kernel: Call Trace: Sep 15 04:36:16 smtp2 snmpd[2489]: Connection from UDP: [206.16.46.12]:59774 Sep 15 04:36:16 smtp2 kernel: [<ffffffff800c3bda>] out_of_memory+0x8e/0x2f3 Sep 15 04:36:16 smtp2 snmpd[2489]: Connection from UDP: [206.16.46.12]:59774 Sep 15 04:36:16 smtp2 kernel: [<ffffffff8000f2ea>] __alloc_pages+0x245/0x2ce Sep 15 04:36:16 smtp2 snmpd[2489]: Connection from UDP: [206.16.46.12]:59774 Sep 15 04:36:16 smtp2 kernel: [<ffffffff8001748c>] cache_grow+0x133/0x381 Sep 15 04:36:16 smtp2 snmpd[2489]: Connection from UDP: [206.16.46.12]:59774 Sep 15 04:36:16 smtp2 kernel: [<ffffffff8005bc74>] cache_alloc_refill+0x136/0x186 Sep 15 04:36:16 smtp2 snmpd[2489]: Connection from UDP: [206.16.46.12]:59774 Sep 15 04:36:16 smtp2 kernel: [<ffffffff800d6fbf>] __kmalloc+0x95/0x9f Sep 15 04:36:16 smtp2 snmpd[2489]: Connection from UDP: [206.16.46.12]:59774 Sep 15 04:36:16 smtp2 kernel: [<ffffffff8002deee>] __alloc_skb+0x5c/0x122 Sep 15 04:36:16 smtp2 snmpd[2489]: Connection from UDP: [206.16.46.12]:59774 Sep 15 04:36:16 smtp2 kernel: [<ffffffff80216292>] sock_alloc_send_skb+0x74/0x1dc Sep 15 04:36:16 smtp2 snmpd[2489]: Connection from UDP: [206.16.46.12]:59774 Sep 15 04:36:16 smtp2 kernel: [<ffffffff80063098>] thread_return+0x62/0xfe Sep 15 04:36:16 smtp2 snmpd[2489]: Connection from UDP: [206.16.46.12]:59774 Sep 15 04:36:16 smtp2 kernel: [<ffffffff80268e42>] unix_dgram_sendmsg+0x17f/0x4e0 Sep 15 04:36:16 smtp2 snmpd[2489]: Connection from UDP: [206.16.46.12]:59774 Sep 15 04:36:16 smtp2 kernel: [<ffffffff80054ab8>] sock_sendmsg+0xf3/0x110 Sep 15 04:36:16 smtp2 snmpd[2489]: Connection from UDP: [206.16.46.12]:59774 Sep 15 04:36:16 smtp2 kernel: [<ffffffff8000e87a>] link_path_walk+0xd3/0xe5 Sep 15 04:36:16 smtp2 snmpd[2489]: Connection from UDP: [206.16.46.12]:59774 Sep 15 04:36:16 smtp2 kernel: [<ffffffff8009dbae>] autoremove_wake_function+0x0/0x2e Sep 15 04:36:16 smtp2 snmpd[2489]: Connection from UDP: [206.16.46.12]:59774 Sep 15 04:36:16 smtp2 kernel: [<ffffffff8000c394>] _atomic_dec_and_lock+0x39/0x57 Sep 15 04:36:16 smtp2 snmpd[2489]: Connection from UDP: [206.16.46.12]:59774 Sep 15 04:36:16 smtp2 kernel: [<ffffffff8002c7f5>] mntput_no_expire+0x19/0x89 Sep 15 04:36:16 smtp2 snmpd[2489]: Connection from UDP: [206.16.46.12]:59774 Sep 15 04:36:16 smtp2 kernel: [<ffffffff80214e4d>] sys_sendto+0x11c/0x14f Sep 15 04:36:16 smtp2 snmpd[2489]: Connection from UDP: [206.16.46.12]:59774 Sep 15 04:36:16 smtp2 kernel: [<ffffffff8005d28d>] tracesys+0xd5/0xe0 Sep 15 04:36:17 smtp2 snmpd[2489]: Connection from UDP: [206.16.46.12]:59774 Sep 15 04:36:17 smtp2 kernel: Sep 15 04:36:17 smtp2 kernel: Mem-info: Sep 15 04:36:17 smtp2 kernel: Node 0 DMA per-cpu: Sep 15 04:36:17 smtp2 kernel: cpu 0 hot: high 0, batch 1 used:0 Sep 15 04:36:17 smtp2 kernel: cpu 0 cold: high 0, batch 1 used:0 Sep 15 04:36:18 smtp2 snmpd[2489]: Connection from UDP: [206.16.46.12]:59774 Sep 15 04:36:18 smtp2 kernel: cpu 1 hot: high 0, batch 1 used:0 Sep 15 04:36:18 smtp2 snmpd[2489]: Connection from UDP: [206.16.46.12]:59774 Sep 15 04:36:18 smtp2 kernel: cpu 1 cold: high 0, batch 1 used:0 Sep 15 04:36:18 smtp2 snmpd[2489]: Connection from UDP: [206.16.46.12]:59774 Sep 15 04:36:18 smtp2 kernel: Node 0 DMA32 per-cpu: Sep 15 04:36:18 smtp2 snmpd[2489]: Connection from UDP: [206.16.46.12]:59774 Sep 15 04:36:18 smtp2 kernel: cpu 0 hot: high 186, batch 31 used:69 Sep 15 04:36:18 smtp2 snmpd[2489]: Connection from UDP: [206.16.46.12]:59774 Sep 15 04:36:18 smtp2 kernel: cpu 0 cold: high 62, batch 15 used:0 Sep 15 04:36:18 smtp2 snmpd[2489]: Connection from UDP: [206.16.46.12]:59774 Sep 15 04:36:18 smtp2 kernel: cpu 1 hot: high 186, batch 31 used:43 Sep 15 04:36:18 smtp2 snmpd[2489]: Connection from UDP: [206.16.46.12]:59774 Sep 15 04:36:18 smtp2 kernel: cpu 1 cold: high 62, batch 15 used:13 Sep 15 04:36:18 smtp2 snmpd[2489]: Connection from UDP: [206.16.46.12]:59774 Sep 15 04:36:18 smtp2 kernel: Node 0 Normal per-cpu: empty Sep 15 04:36:18 smtp2 snmpd[2489]: Connection from UDP: [206.16.46.12]:59774 Sep 15 04:36:18 smtp2 kernel: Node 0 HighMem per-cpu: empty Sep 15 04:36:19 smtp2 snmpd[2489]: Connection from UDP: [206.16.46.12]:59774 Sep 15 04:36:20 smtp2 kernel: Free pages: 8132kB (0kB HighMem) Sep 15 04:36:20 smtp2 snmpd[2489]: Connection from UDP: [206.16.46.12]:59774 Sep 15 04:36:22 smtp2 kernel: Active:195980 inactive:159056 dirty:0 writeback:0 unstable:0 free:2033 slab:32659 mapped-file:2736 mapped-anon:353032 pagetables:97169 Sep 15 04:36:24 smtp2 kernel: Node 0 DMA free:2152kB min:28kB low:32kB high:40kB active:0kB inactive:0kB present:10716kB pages_scanned:0 all_unreclaimable? yes Sep 15 04:36:39 smtp2 kernel: lowmem_reserve[]: 0 2003 2003 2003 Sep 15 04:37:20 smtp2 kernel: Node 0 DMA32 free:5980kB min:5708kB low:7132kB high:8560kB active:783924kB inactive:636220kB present:2051184kB pages_scanned:3003394 all_unreclaimable? yes Sep 15 04:37:20 smtp2 kernel: lowmem_reserve[]: 0 0 0 0 Sep 15 04:37:20 smtp2 kernel: Node 0 Normal free:0kB min:0kB low:0kB high:0kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no Sep 15 04:37:20 smtp2 kernel: lowmem_reserve[]: 0 0 0 0 Sep 15 04:37:20 smtp2 kernel: Node 0 HighMem free:0kB min:128kB low:128kB high:128kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no Sep 15 04:37:20 smtp2 kernel: lowmem_reserve[]: 0 0 0 0 Sep 15 04:37:20 smtp2 kernel: Node 0 DMA: 2*4kB 4*8kB 2*16kB 5*32kB 4*64kB 1*128kB 2*256kB 0*512kB 1*1024kB 0*2048kB 0*4096kB = 2152kB Sep 15 04:37:20 smtp2 kernel: Node 0 DMA32: 87*4kB 34*8kB 3*16kB 12*32kB 5*64kB 2*128kB 1*256kB 0*512kB 0*1024kB 0*2048kB 1*4096kB = 5980kB Sep 15 04:37:20 smtp2 kernel: Node 0 Normal: empty Sep 15 04:37:20 smtp2 kernel: Node 0 HighMem: empty Sep 15 04:37:20 smtp2 kernel: 6591 pagecache pages Sep 15 04:37:20 smtp2 kernel: Swap cache: add 619164, delete 615423, find 100105/111253, race 3+10 Sep 15 04:37:20 smtp2 kernel: Free swap = 0kB Sep 15 04:37:20 smtp2 kernel: Total swap = 2096472kB Sep 15 04:37:20 smtp2 kernel: Free swap: 0kB Sep 15 04:37:20 smtp2 kernel: 524000 pages of RAM Sep 15 04:37:20 smtp2 kernel: 9347 reserved pages Sep 15 04:37:20 smtp2 kernel: 1055448 pages shared Sep 15 04:37:20 smtp2 kernel: 3741 pages swap cached Sep 15 04:37:20 smtp2 kernel: Out of memory: Killed process 3431 (crond). Sep 15 04:38:12 smtp2 snmpd[2489]: Connection from UDP: [206.16.46.12]:59774 Sep 15 04:38:14 smtp2 last message repeated 44 times Sep 15 04:38:14 smtp2 snmpd[2489]: Connection from UDP: [206.16.46.12]:53589 Sep 15 04:38:14 smtp2 snmpd[2489]: Received SNMP packet(s) from UDP: [206.16.46.12]:53589 Sep 15 04:38:14 smtp2 snmpd[2489]: Connection from UDP: [206.16.46.12]:53589 Sep 15 04:38:14 smtp2 last message repeated 130 times Sep 15 04:38:14 smtp2 snmpd[2489]: Connection from UDP: [206.16.46.12]:44340 Sep 15 04:38:14 smtp2 snmpd[2489]: Received SNMP packet(s) from UDP: [206.16.46.12]:44340 Sep 15 04:38:14 smtp2 snmpd[2489]: Connection from UDP: [206.16.46.12]:44340 Sep 15 04:38:14 smtp2 last message repeated 113 times Sep 15 04:40:43 smtp2 kernel: modclusterd invoked oom-killer: gfp_mask=0x201d2, order=0, oomkilladj=0 Sep 15 04:40:43 smtp2 kernel: Sep 15 04:40:43 smtp2 kernel: Call Trace: Sep 15 04:40:43 smtp2 kernel: [<ffffffff800c3bda>] out_of_memory+0x8e/0x2f3 Sep 15 04:40:43 smtp2 kernel: [<ffffffff8000f2ea>] __alloc_pages+0x245/0x2ce Sep 15 04:40:43 smtp2 kernel: [<ffffffff80012a77>] __do_page_cache_readahead+0xa2/0x1e6 Sep 15 04:40:43 smtp2 kernel: [<ffffffff80063a05>] __wait_on_bit_lock+0x5b/0x66 Sep 15 04:40:43 smtp2 kernel: [<ffffffff8003f626>] __lock_page+0x5e/0x64 Sep 15 04:40:43 smtp2 kernel: [<ffffffff80013406>] filemap_nopage+0x148/0x322 Sep 15 04:40:43 smtp2 kernel: [<ffffffff8000886a>] __handle_mm_fault+0x1f8/0xe5c Sep 15 04:40:43 smtp2 kernel: [<ffffffff80066b9f>] do_page_fault+0x4cb/0x830 Sep 15 04:40:43 smtp2 kernel: [<ffffffff800320b8>] sys_faccessat+0xe4/0x18d Sep 15 04:40:43 smtp2 kernel: [<ffffffff8005dde9>] error_exit+0x0/0x84 Sep 15 04:40:43 smtp2 kernel: Sep 15 04:40:43 smtp2 kernel: Mem-info: Sep 15 04:40:43 smtp2 kernel: Node 0 DMA per-cpu: Sep 15 04:40:43 smtp2 kernel: cpu 0 hot: high 0, batch 1 used:0 Sep 15 04:40:43 smtp2 kernel: cpu 0 cold: high 0, batch 1 used:0 Sep 15 04:40:43 smtp2 kernel: cpu 1 hot: high 0, batch 1 used:0 Sep 15 04:40:43 smtp2 kernel: cpu 1 cold: high 0, batch 1 used:0 Sep 15 04:40:43 smtp2 kernel: Node 0 DMA32 per-cpu: Sep 15 04:40:43 smtp2 kernel: cpu 0 hot: high 186, batch 31 used:35 Sep 15 04:40:43 smtp2 kernel: cpu 0 cold: high 62, batch 15 used:4 Sep 15 04:40:43 smtp2 kernel: cpu 1 hot: high 186, batch 31 used:43 Sep 15 04:40:43 smtp2 kernel: cpu 1 cold: high 62, batch 15 used:14 Sep 15 04:40:43 smtp2 kernel: Node 0 Normal per-cpu: empty Sep 15 04:40:43 smtp2 kernel: Node 0 HighMem per-cpu: empty Sep 15 04:40:43 smtp2 kernel: Free pages: 7804kB (0kB HighMem) Sep 15 04:40:43 smtp2 kernel: Active:144060 inactive:212032 dirty:7 writeback:0 unstable:0 free:1951 slab:32604 mapped-file:2742 mapped-anon:353322 pagetables:97083 Sep 15 04:40:43 smtp2 kernel: Node 0 DMA free:2152kB min:28kB low:32kB high:40kB active:0kB inactive:0kB present:10716kB pages_scanned:0 all_unreclaimable? yes Sep 15 04:40:43 smtp2 kernel: lowmem_reserve[]: 0 2003 2003 2003 Sep 15 04:40:43 smtp2 kernel: Node 0 DMA32 free:5652kB min:5708kB low:7132kB high:8560kB active:576240kB inactive:848128kB present:2051184kB pages_scanned:2766187 all_unreclaimable? yes Sep 15 04:40:43 smtp2 kernel: lowmem_reserve[]: 0 0 0 0 Sep 15 04:40:43 smtp2 kernel: Node 0 Normal free:0kB min:0kB low:0kB high:0kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no Sep 15 04:40:43 smtp2 kernel: lowmem_reserve[]: 0 0 0 0 Sep 15 04:40:43 smtp2 kernel: Node 0 HighMem free:0kB min:128kB low:128kB high:128kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no Sep 15 04:40:43 smtp2 kernel: lowmem_reserve[]: 0 0 0 0 Sep 15 04:40:43 smtp2 kernel: Node 0 DMA: 2*4kB 4*8kB 2*16kB 5*32kB 4*64kB 1*128kB 2*256kB 0*512kB 1*1024kB 0*2048kB 0*4096kB = 2152kB Sep 15 04:40:43 smtp2 kernel: Node 0 DMA32: 17*4kB 24*8kB 5*16kB 12*32kB 5*64kB 2*128kB 1*256kB 0*512kB 0*1024kB 0*2048kB 1*4096kB = 5652kB Sep 15 04:40:43 smtp2 kernel: Node 0 Normal: empty Sep 15 04:40:43 smtp2 kernel: Node 0 HighMem: empty Sep 15 04:40:43 smtp2 kernel: 6698 pagecache pages Sep 15 04:40:43 smtp2 kernel: Swap cache: add 626883, delete 623182, find 100385/112417, race 3+15 Sep 15 04:40:43 smtp2 kernel: Free swap = 0kB Sep 15 04:40:43 smtp2 kernel: Total swap = 2096472kB Sep 15 04:40:43 smtp2 kernel: Free swap: 0kB Sep 15 04:40:43 smtp2 kernel: 524000 pages of RAM Sep 15 04:40:43 smtp2 kernel: 9347 reserved pages Sep 15 04:40:43 smtp2 kernel: 1056040 pages shared Sep 15 04:40:43 smtp2 kernel: 3701 pages swap cached Sep 15 04:40:43 smtp2 kernel: Out of memory: Killed process 3449 (crond). Sep 15 04:40:52 smtp2 snmpd[2489]: Connection from UDP: [206.16.46.12]:50576 Sep 15 04:40:52 smtp2 snmpd[2489]: Received SNMP packet(s) from UDP: [206.16.46.12]:50576 Sep 15 04:40:53 smtp2 snmpd[2489]: Connection from UDP: [206.16.46.12]:50576 Sep 15 04:40:53 smtp2 last message repeated 60 times Sep 15 04:43:46 smtp2 kernel: postdrop invoked oom-killer: gfp_mask=0x201d2, order=0, oomkilladj=0 Sep 15 04:43:56 smtp2 kernel: Sep 15 04:43:56 smtp2 kernel: Call Trace: Sep 15 04:43:56 smtp2 kernel: [<ffffffff800c3bda>] out_of_memory+0x8e/0x2f3 Sep 15 04:43:56 smtp2 kernel: [<ffffffff8000f2ea>] __alloc_pages+0x245/0x2ce Sep 15 04:43:56 smtp2 kernel: [<ffffffff80012a77>] __do_page_cache_readahead+0xa2/0x1e6 Sep 15 04:43:56 smtp2 kernel: [<ffffffff80063a05>] __wait_on_bit_lock+0x5b/0x66 Sep 15 04:43:56 smtp2 kernel: [<ffffffff8003f626>] __lock_page+0x5e/0x64 Sep 15 04:43:56 smtp2 kernel: [<ffffffff80013406>] filemap_nopage+0x148/0x322 Sep 15 04:43:56 smtp2 kernel: [<ffffffff8000886a>] __handle_mm_fault+0x1f8/0xe5c Sep 15 04:43:56 smtp2 kernel: [<ffffffff80066b9f>] do_page_fault+0x4cb/0x830 Sep 15 04:43:56 smtp2 kernel: [<ffffffff80059fdf>] hrtimer_cancel+0xc/0x16 Sep 15 04:43:56 smtp2 kernel: [<ffffffff80063db7>] do_nanosleep+0x47/0x70 Sep 15 04:43:57 smtp2 kernel: [<ffffffff80059ecc>] hrtimer_nanosleep+0x58/0x118 Sep 15 04:43:57 smtp2 kernel: [<ffffffff8005dde9>] error_exit+0x0/0x84 Sep 15 04:43:57 smtp2 kernel: Sep 15 04:43:57 smtp2 kernel: Mem-info: Sep 15 04:43:57 smtp2 kernel: Node 0 DMA per-cpu: Sep 15 04:43:57 smtp2 kernel: cpu 0 hot: high 0, batch 1 used:0 Sep 15 04:43:57 smtp2 kernel: cpu 0 cold: high 0, batch 1 used:0 Sep 15 04:43:57 smtp2 kernel: cpu 1 hot: high 0, batch 1 used:0 Sep 15 04:43:57 smtp2 kernel: cpu 1 cold: high 0, batch 1 used:0 Sep 15 04:43:57 smtp2 kernel: Node 0 DMA32 per-cpu: Sep 15 04:43:57 smtp2 kernel: cpu 0 hot: high 186, batch 31 used:41 Sep 15 04:43:57 smtp2 kernel: cpu 0 cold: high 62, batch 15 used:60 Sep 15 04:43:57 smtp2 kernel: cpu 1 hot: high 186, batch 31 used:14 Sep 15 04:43:57 smtp2 kernel: cpu 1 cold: high 62, batch 15 used:60 Sep 15 04:43:57 smtp2 kernel: Node 0 Normal per-cpu: empty Sep 15 04:43:58 smtp2 kernel: Node 0 HighMem per-cpu: empty Sep 15 04:43:58 smtp2 kernel: Free pages: 7816kB (0kB HighMem) Sep 15 04:43:58 smtp2 kernel: Active:224553 inactive:131064 dirty:1 writeback:0 unstable:0 free:1954 slab:32705 mapped-file:2719 mapped-anon:353434 pagetables:96958 Sep 15 04:43:58 smtp2 kernel: Node 0 DMA free:2152kB min:28kB low:32kB high:40kB active:0kB inactive:0kB present:10716kB pages_scanned:0 all_unreclaimable? yes Sep 15 04:43:58 smtp2 kernel: lowmem_reserve[]: 0 2003 2003 2003 Sep 15 04:43:58 smtp2 kernel: Node 0 DMA32 free:5664kB min:5708kB low:7132kB high:8560kB active:898120kB inactive:524348kB present:2051184kB pages_scanned:2946614 all_unreclaimable? yes Sep 15 04:43:58 smtp2 kernel: lowmem_reserve[]: 0 0 0 0 Sep 15 04:43:58 smtp2 kernel: Node 0 Normal free:0kB min:0kB low:0kB high:0kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no Sep 15 04:43:58 smtp2 kernel: lowmem_reserve[]: 0 0 0 0 Sep 15 04:43:58 smtp2 kernel: Node 0 HighMem free:0kB min:128kB low:128kB high:128kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no Sep 15 04:43:58 smtp2 kernel: lowmem_reserve[]: 0 0 0 0 Sep 15 04:43:58 smtp2 kernel: Node 0 DMA: 2*4kB 4*8kB 2*16kB 5*32kB 4*64kB 1*128kB 2*256kB 0*512kB 1*1024kB 0*2048kB 0*4096kB = 2152kB Sep 15 04:43:58 smtp2 kernel: Node 0 DMA32: 14*4kB 27*8kB 5*16kB 12*32kB 5*64kB 2*128kB 1*256kB 0*512kB 0*1024kB 0*2048kB 1*4096kB = 5664kB Sep 15 04:43:58 smtp2 kernel: Node 0 Normal: empty Sep 15 04:43:58 smtp2 kernel: Node 0 HighMem: empty Sep 15 04:43:58 smtp2 kernel: 6522 pagecache pages Sep 15 04:43:58 smtp2 kernel: Swap cache: add 630480, delete 626780, find 100415/112836, race 3+15 Sep 15 04:43:58 smtp2 kernel: Free swap = 0kB Sep 15 04:43:58 smtp2 kernel: Total swap = 2096472kB Sep 15 04:43:58 smtp2 kernel: Free swap: 0kB Sep 15 04:43:58 smtp2 kernel: 524000 pages of RAM Sep 15 04:43:58 smtp2 kernel: 9347 reserved pages Sep 15 04:43:58 smtp2 kernel: 1052953 pages shared Sep 15 04:43:58 smtp2 kernel: 3700 pages swap cached Sep 15 04:43:58 smtp2 kernel: Out of memory: Killed process 3467 (crond). Sep 15 04:44:38 smtp2 kernel: postdrop invoked oom-killer: gfp_mask=0x201d2, order=0, oomkilladj=0 Sep 15 04:45:24 smtp2 snmpd[2489]: Connection from UDP: [206.16.46.12]:43992 Sep 15 04:45:24 smtp2 kernel: Sep 15 04:45:25 smtp2 kernel: Call Trace: Sep 15 04:45:25 smtp2 kernel: [<ffffffff800c3bda>] out_of_memory+0x8e/0x2f3 Sep 15 04:45:25 smtp2 kernel: [<ffffffff8000f2ea>] __alloc_pages+0x245/0x2ce Sep 15 04:45:25 smtp2 kernel: [<ffffffff80012a77>] __do_page_cache_readahead+0xa2/0x1e6 Sep 15 04:45:25 smtp2 kernel: [<ffffffff80054ab8>] sock_sendmsg+0xf3/0x110 Sep 15 04:45:25 smtp2 kernel: [<ffffffff8000e87a>] link_path_walk+0xd3/0xe5 Sep 15 04:45:25 smtp2 kernel: [<ffffffff80013406>] filemap_nopage+0x148/0x322 Sep 15 04:45:25 smtp2 kernel: [<ffffffff8000886a>] __handle_mm_fault+0x1f8/0xe5c Sep 15 04:45:25 smtp2 kernel: [<ffffffff80066b9f>] do_page_fault+0x4cb/0x830 Sep 15 04:45:25 smtp2 kernel: [<ffffffff80059fdf>] hrtimer_cancel+0xc/0x16 Sep 15 04:45:25 smtp2 kernel: [<ffffffff80063db7>] do_nanosleep+0x47/0x70 Sep 15 04:45:25 smtp2 kernel: [<ffffffff80059ecc>] hrtimer_nanosleep+0x58/0x118 Sep 15 04:45:25 smtp2 kernel: [<ffffffff8005dde9>] error_exit+0x0/0x84 Sep 15 04:45:25 smtp2 kernel: Sep 15 04:45:25 smtp2 kernel: Mem-info: Sep 15 04:45:25 smtp2 kernel: Node 0 DMA per-cpu: Sep 15 04:45:25 smtp2 kernel: cpu 0 hot: high 0, batch 1 used:0 Sep 15 04:45:25 smtp2 kernel: cpu 0 cold: high 0, batch 1 used:0 Sep 15 04:45:25 smtp2 kernel: cpu 1 hot: high 0, batch 1 used:0 Sep 15 04:45:25 smtp2 kernel: cpu 1 cold: high 0, batch 1 used:0 Sep 15 04:45:25 smtp2 kernel: Node 0 DMA32 per-cpu: Sep 15 04:45:25 smtp2 kernel: cpu 0 hot: high 186, batch 31 used:16 Sep 15 04:45:25 smtp2 kernel: cpu 0 cold: high 62, batch 15 used:14 Sep 15 04:45:25 smtp2 kernel: cpu 1 hot: high 186, batch 31 used:12 Sep 15 04:45:25 smtp2 kernel: cpu 1 cold: high 62, batch 15 used:40 Sep 15 04:45:25 smtp2 kernel: Node 0 Normal per-cpu: empty Sep 15 04:45:25 smtp2 kernel: Node 0 HighMem per-cpu: empty Sep 15 04:45:25 smtp2 kernel: Free pages: 8072kB (0kB HighMem) Sep 15 04:45:25 smtp2 kernel: Active:196819 inactive:156715 dirty:0 writeback:0 unstable:0 free:2018 slab:32583 mapped-file:2736 mapped-anon:353474 pagetables:96986 Sep 15 04:45:25 smtp2 kernel: Node 0 DMA free:2152kB min:28kB low:32kB high:40kB active:0kB inactive:0kB present:10716kB pages_scanned:0 all_unreclaimable? yes Sep 15 04:45:25 smtp2 kernel: lowmem_reserve[]: 0 2003 2003 2003 Sep 15 04:45:25 smtp2 kernel: Node 0 DMA32 free:5920kB min:5708kB low:7132kB high:8560kB active:787276kB inactive:626860kB present:2051184kB pages_scanned:3225864 all_unreclaimable? yes Sep 15 04:45:25 smtp2 kernel: lowmem_reserve[]: 0 0 0 0 Sep 15 04:45:25 smtp2 kernel: Node 0 Normal free:0kB min:0kB low:0kB high:0kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no Sep 15 04:45:25 smtp2 kernel: lowmem_reserve[]: 0 0 0 0 Sep 15 04:45:25 smtp2 kernel: Node 0 HighMem free:0kB min:128kB low:128kB high:128kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no Sep 15 04:45:25 smtp2 kernel: lowmem_reserve[]: 0 0 0 0 Sep 15 04:45:25 smtp2 kernel: Node 0 DMA: 2*4kB 4*8kB 2*16kB 5*32kB 4*64kB 1*128kB 2*256kB 0*512kB 1*1024kB 0*2048kB 0*4096kB = 2152kB Sep 15 04:45:25 smtp2 kernel: Node 0 DMA32: 80*4kB 26*8kB 5*16kB 12*32kB 5*64kB 2*128kB 1*256kB 0*512kB 0*1024kB 0*2048kB 1*4096kB = 5920kB Sep 15 04:45:25 smtp2 kernel: Node 0 Normal: empty Sep 15 04:45:25 smtp2 kernel: Node 0 HighMem: empty Sep 15 04:45:25 smtp2 kernel: 6528 pagecache pages Sep 15 04:45:25 smtp2 kernel: Swap cache: add 630994, delete 627294, find 100446/112908, race 3+15 Sep 15 04:45:25 smtp2 kernel: Free swap = 0kB Sep 15 04:45:25 smtp2 kernel: Total swap = 2096472kB Sep 15 04:45:25 smtp2 kernel: Free swap: 0kB Sep 15 04:45:25 smtp2 kernel: 524000 pages of RAM Sep 15 04:45:25 smtp2 kernel: 9347 reserved pages Sep 15 04:45:25 smtp2 kernel: 1055739 pages shared Sep 15 04:45:25 smtp2 kernel: 3700 pages swap cached Andrea Dell'Amico wrote: > On Wed, 2009-09-16 at 03:23 -0700, David Buechler wrote: > >> Hi Andrea! >> >> Heh, yeah, I should have specified that. No, I'm not using Xen on this >> particular system. I'm using Xen on other systems under DRBD 8.2 and >> they're pretty stable. >> > > Ugh. A new kind of crash, not reassuring :-). > I think you should post the kernel Oops on the list, then. > > >> -- >> Regards, >> David A. Buechler >> System Administrator, >> CableAmerica Missouri >> > > ciao > andrea > > -- This message has been scanned for viruses and dangerous content by MailScanner, and is believed to be clean.