[DRBD-user] Crash - Centos 5.3, drbd83 RPM

Dave Buechler dbuechler at cablemo.com
Fri Sep 18 00:29:08 CEST 2009

Note: "permalinks" may not be as permanent as we would like,
direct links of old sources may well be a few messages off.


Memo to Self:  Self, try Reply-All if you want more than one person to 
look at your problem. ;-)

Original message:


Ok.  I have six hours of log data, but it all says pretty much the same
thing.  Here's the first 15 minutes or so.  If you want more, I'll send
more.

Thanks for looking at this.

-- 
Regards,
David A. Buechler
System Administrator,
CableAmerica Missouri



/var/log/messages:
Sep 15 04:17:38 smtp2 heartbeat: [2717]: WARN: Gmain_timeout_dispatch:
Dispatch function for send local status took too long to execute: 150 ms
(> 50 ms) (GSource: 0x1aacc9f8)
Sep 15 04:19:38 smtp2 snmpd[2489]: Connection from UDP:
[206.16.46.12]:46136
Sep 15 04:19:38 smtp2 snmpd[2489]: Received SNMP packet(s) from UDP:
[206.16.46.12]:46136
Sep 15 04:19:42 smtp2 snmpd[2489]: Connection from UDP:
[206.16.46.12]:46136
Sep 15 04:19:49 smtp2 last message repeated 35 times
Sep 15 04:31:41 smtp2 kernel: block drbd0: sock was shut down by peer
Sep 15 04:31:42 smtp2 kernel: block drbd0: peer( Primary -> Unknown )
conn( Connected -> BrokenPipe ) pdsk( UpToDate -> DUnknown )
Sep 15 04:31:42 smtp2 kernel: block drbd0: short read expecting header
on sock: r=0
Sep 15 04:31:42 smtp2 kernel: block drbd0: md_sync_timer expired! Worker
calls drbd_md_sync().
Sep 15 04:31:42 smtp2 kernel: block drbd0: msock_sendmsg returned -32
Sep 15 04:31:42 smtp2 kernel: block drbd0: short sent PingAck size=8 sent=0
Sep 15 04:31:42 smtp2 kernel: block drbd0: asender terminated
Sep 15 04:31:42 smtp2 kernel: block drbd0: Terminating asender thread
Sep 15 04:31:42 smtp2 kernel: block drbd0: Connection closed
Sep 15 04:31:42 smtp2 kernel: block drbd0: conn( BrokenPipe ->
Unconnected )
Sep 15 04:31:42 smtp2 kernel: block drbd0: receiver terminated
Sep 15 04:31:42 smtp2 kernel: block drbd0: Restarting receiver thread
Sep 15 04:31:42 smtp2 kernel: block drbd0: receiver (re)started
Sep 15 04:31:42 smtp2 kernel: block drbd0: conn( Unconnected ->
WFConnection )
Sep 15 04:31:42 smtp2 kernel: block drbd0: Handshake successful: Agreed
network protocol version 90
Sep 15 04:31:42 smtp2 kernel: block drbd0: conn( WFConnection ->
WFReportParams )
Sep 15 04:31:42 smtp2 kernel: block drbd0: Starting asender thread (from
drbd0_receiver [2581])
Sep 15 04:31:42 smtp2 kernel: block drbd0: data-integrity-alg: <not-used>
Sep 15 04:31:42 smtp2 kernel: block drbd0: drbd_sync_handshake:
Sep 15 04:31:42 smtp2 kernel: block drbd0: self
E6B7DB61310BE3A6:0000000000000000:B939FB24AD00D552:7E8DF909A8E37EDF
bits:0 flags:0
Sep 15 04:31:42 smtp2 kernel: block drbd0: peer
E720051EDAD32591:E6B7DB61310BE3A7:B939FB24AD00D552:7E8DF909A8E37EDF
bits:1090 flags:0
Sep 15 04:31:42 smtp2 kernel: block drbd0: uuid_compare()=-1 by rule 5
Sep 15 04:31:42 smtp2 kernel: block drbd0: peer( Unknown -> Primary )
conn( WFReportParams -> WFBitMapT ) pdsk( DUnknown -> UpToDate )
Sep 15 04:31:42 smtp2 kernel: block drbd0: conn( WFBitMapT -> WFSyncUUID )
Sep 15 04:31:42 smtp2 kernel: block drbd0: helper command: /sbin/drbdadm
before-resync-target minor-0
Sep 15 04:31:42 smtp2 kernel: block drbd0: md_sync_timer expired! Worker
calls drbd_md_sync().
Sep 15 04:31:42 smtp2 kernel: block drbd0: helper command: /sbin/drbdadm
before-resync-target minor-0 exit code 0 (0x0)
Sep 15 04:31:42 smtp2 kernel: block drbd0: conn( WFSyncUUID ->
SyncTarget ) disk( UpToDate -> Inconsistent )
Sep 15 04:31:42 smtp2 kernel: block drbd0: Began resync as SyncTarget
(will sync 4360 KB [1090 bits set]).
Sep 15 04:31:42 smtp2 kernel: block drbd0: Resync done (total 3 sec;
paused 0 sec; 1452 K/sec)
Sep 15 04:31:42 smtp2 kernel: block drbd0: conn( SyncTarget -> Connected
) disk( Inconsistent -> UpToDate )
Sep 15 04:31:42 smtp2 kernel: block drbd0: helper command: /sbin/drbdadm
after-resync-target minor-0
Sep 15 04:31:42 smtp2 kernel: drbdadm invoked oom-killer:
gfp_mask=0x84d0, order=0, oomkilladj=0
Sep 15 04:31:42 smtp2 kernel:
Sep 15 04:31:42 smtp2 kernel: Call Trace:
Sep 15 04:31:42 smtp2 kernel:  [<ffffffff800c3bda>] out_of_memory+0x8e/0x2f3
Sep 15 04:31:42 smtp2 kernel:  [<ffffffff8000f2ea>]
__alloc_pages+0x245/0x2ce
Sep 15 04:31:42 smtp2 kernel:  [<ffffffff8002309f>]
alloc_page_interleave+0x3d/0x74
Sep 15 04:31:42 smtp2 kernel:  [<ffffffff8002b5e5>]
get_zeroed_page+0x21/0x82
Sep 15 04:31:42 smtp2 kernel:  [<ffffffff8002d80c>] __pmd_alloc+0x14/0x8c
Sep 15 04:31:42 smtp2 kernel:  [<ffffffff80008127>]
copy_page_range+0x1f3/0x73e
Sep 15 04:31:42 smtp2 kernel:  [<ffffffff800d6f0e>]
alternate_node_alloc+0x70/0x8c
Sep 15 04:31:42 smtp2 kernel:  [<ffffffff8001f847>]
copy_process+0xd30/0x15b8
Sep 15 04:31:42 smtp2 kernel:  [<ffffffff80030d3a>] do_fork+0x69/0x1c1
Sep 15 04:31:42 smtp2 kernel:  [<ffffffff8005d28d>] tracesys+0xd5/0xe0
Sep 15 04:31:42 smtp2 kernel:  [<ffffffff8005d427>]
ptregscall_common+0x67/0xac
Sep 15 04:31:42 smtp2 kernel:
Sep 15 04:31:42 smtp2 kernel: Mem-info:
Sep 15 04:31:42 smtp2 kernel: Node 0 DMA per-cpu:
Sep 15 04:31:42 smtp2 kernel: cpu 0 hot: high 0, batch 1 used:0
Sep 15 04:31:42 smtp2 kernel: cpu 0 cold: high 0, batch 1 used:0
Sep 15 04:31:42 smtp2 kernel: cpu 1 hot: high 0, batch 1 used:0
Sep 15 04:31:42 smtp2 kernel: cpu 1 cold: high 0, batch 1 used:0
Sep 15 04:31:42 smtp2 kernel: Node 0 DMA32 per-cpu:
Sep 15 04:31:42 smtp2 kernel: cpu 0 hot: high 186, batch 31 used:48
Sep 15 04:31:42 smtp2 kernel: cpu 0 cold: high 62, batch 15 used:40
Sep 15 04:31:42 smtp2 kernel: cpu 1 hot: high 186, batch 31 used:156
Sep 15 04:31:42 smtp2 kernel: cpu 1 cold: high 62, batch 15 used:60
Sep 15 04:31:42 smtp2 kernel: Node 0 Normal per-cpu: empty
Sep 15 04:31:42 smtp2 kernel: Node 0 HighMem per-cpu: empty
Sep 15 04:31:42 smtp2 kernel: Free pages:        7860kB (0kB HighMem)
Sep 15 04:31:42 smtp2 kernel: Active:222971 inactive:131062 dirty:0
writeback:0 unstable:0 free:1965 slab:32477 mapped-file:2705
mapped-anon:353176 pagetables:97307
Sep 15 04:31:42 smtp2 kernel: Node 0 DMA free:2152kB min:28kB low:32kB
high:40kB active:0kB inactive:0kB present:10716kB pages_scanned:0
all_unreclaimable? yes
Sep 15 04:31:42 smtp2 kernel: lowmem_reserve[]: 0 2003 2003 2003
Sep 15 04:31:42 smtp2 kernel: Node 0 DMA32 free:5708kB min:5708kB
low:7132kB high:8560kB active:891884kB inactive:524248kB
present:2051184kB pages_scanned:2988074 all_unreclaimable? yes
Sep 15 04:31:42 smtp2 kernel: lowmem_reserve[]: 0 0 0 0
Sep 15 04:31:42 smtp2 kernel: Node 0 Normal free:0kB min:0kB low:0kB
high:0kB active:0kB inactive:0kB present:0kB pages_scanned:0
all_unreclaimable? no
Sep 15 04:31:42 smtp2 kernel: lowmem_reserve[]: 0 0 0 0
Sep 15 04:31:42 smtp2 kernel: Node 0 HighMem free:0kB min:128kB
low:128kB high:128kB active:0kB inactive:0kB present:0kB pages_scanned:0
all_unreclaimable? no
Sep 15 04:31:42 smtp2 kernel: lowmem_reserve[]: 0 0 0 0
Sep 15 04:31:42 smtp2 kernel: Node 0 DMA: 2*4kB 4*8kB 2*16kB 5*32kB
4*64kB 1*128kB 2*256kB 0*512kB 1*1024kB 0*2048kB 0*4096kB = 2152kB
Sep 15 04:31:42 smtp2 kernel: Node 0 DMA32: 129*4kB 1*8kB 2*16kB 13*32kB
4*64kB 1*128kB 1*256kB 0*512kB 0*1024kB 0*2048kB 1*4096kB = 5708kB
Sep 15 04:31:42 smtp2 kernel: Node 0 Normal: empty
Sep 15 04:31:42 smtp2 kernel: Node 0 HighMem: empty
Sep 15 04:31:42 smtp2 kernel: 6436 pagecache pages
Sep 15 04:31:42 smtp2 kernel: Swap cache: add 613470, delete 609770,
find 99957/110522, race 3+8
Sep 15 04:31:42 smtp2 kernel: Free swap  = 0kB
Sep 15 04:31:42 smtp2 kernel: Total swap = 2096472kB
Sep 15 04:31:42 smtp2 kernel: Free swap:            0kB
Sep 15 04:31:42 smtp2 kernel: 524000 pages of RAM
Sep 15 04:31:42 smtp2 kernel: 9347 reserved pages
Sep 15 04:31:42 smtp2 kernel: 1057738 pages shared
Sep 15 04:31:42 smtp2 kernel: 3700 pages swap cached
Sep 15 04:31:42 smtp2 kernel: Out of memory: Killed process 3250 (crond).
Sep 15 04:31:42 smtp2 kernel: postdrop invoked oom-killer:
gfp_mask=0x201d2, order=0, oomkilladj=0
Sep 15 04:31:42 smtp2 kernel:
Sep 15 04:31:42 smtp2 kernel: Call Trace:
Sep 15 04:31:42 smtp2 kernel:  [<ffffffff800c3bda>] out_of_memory+0x8e/0x2f3
Sep 15 04:31:42 smtp2 kernel:  [<ffffffff8000f2ea>]
__alloc_pages+0x245/0x2ce
Sep 15 04:31:42 smtp2 kernel:  [<ffffffff80012a77>]
__do_page_cache_readahead+0xa2/0x1e6
Sep 15 04:31:42 smtp2 kernel:  [<ffffffff80054ab8>] sock_sendmsg+0xf3/0x110
Sep 15 04:31:42 smtp2 kernel:  [<ffffffff8000e87a>] link_path_walk+0xd3/0xe5
Sep 15 04:31:42 smtp2 kernel:  [<ffffffff80013406>]
filemap_nopage+0x148/0x322
Sep 15 04:31:42 smtp2 kernel:  [<ffffffff8000886a>]
__handle_mm_fault+0x1f8/0xe5c
Sep 15 04:31:42 smtp2 kernel:  [<ffffffff80066b9f>]
do_page_fault+0x4cb/0x830
Sep 15 04:31:42 smtp2 kernel:  [<ffffffff80059fdf>] hrtimer_cancel+0xc/0x16
Sep 15 04:31:42 smtp2 kernel:  [<ffffffff80063db7>] do_nanosleep+0x47/0x70
Sep 15 04:31:42 smtp2 kernel:  [<ffffffff80059ecc>]
hrtimer_nanosleep+0x58/0x118
Sep 15 04:31:42 smtp2 kernel:  [<ffffffff8005dde9>] error_exit+0x0/0x84
Sep 15 04:31:42 smtp2 kernel:
Sep 15 04:31:42 smtp2 kernel: Mem-info:
Sep 15 04:31:42 smtp2 kernel: Node 0 DMA per-cpu:
Sep 15 04:31:42 smtp2 kernel: cpu 0 hot: high 0, batch 1 used:0
Sep 15 04:31:42 smtp2 kernel: cpu 0 cold: high 0, batch 1 used:0
Sep 15 04:31:42 smtp2 kernel: cpu 1 hot: high 0, batch 1 used:0
Sep 15 04:31:42 smtp2 kernel: cpu 1 cold: high 0, batch 1 used:0
Sep 15 04:31:42 smtp2 kernel: Node 0 DMA32 per-cpu:
Sep 15 04:31:42 smtp2 kernel: cpu 0 hot: high 186, batch 31 used:57
Sep 15 04:31:42 smtp2 kernel: cpu 0 cold: high 62, batch 15 used:37
Sep 15 04:31:42 smtp2 kernel: cpu 1 hot: high 186, batch 31 used:166
Sep 15 04:31:42 smtp2 kernel: cpu 1 cold: high 62, batch 15 used:14
Sep 15 04:31:42 smtp2 kernel: Node 0 Normal per-cpu: empty
Sep 15 04:31:42 smtp2 kernel: Node 0 HighMem per-cpu: empty
Sep 15 04:31:42 smtp2 kernel: Free pages:        7984kB (0kB HighMem)
Sep 15 04:31:42 smtp2 kernel: Active:182184 inactive:162368 dirty:0
writeback:0 unstable:0 free:1996 slab:32439 mapped-file:2705
mapped-anon:353176 pagetables:97307
Sep 15 04:31:42 smtp2 kernel: Node 0 DMA free:2152kB min:28kB low:32kB
high:40kB active:0kB inactive:0kB present:10716kB pages_scanned:0
all_unreclaimable? yes
Sep 15 04:31:42 smtp2 kernel: lowmem_reserve[]: 0 2003 2003 2003
Sep 15 04:31:42 smtp2 kernel: Node 0 DMA32 free:5832kB min:5708kB
low:7132kB high:8560kB active:728576kB inactive:649760kB
present:2051184kB pages_scanned:422478 all_unreclaimable? no
Sep 15 04:31:42 smtp2 kernel: lowmem_reserve[]: 0 0 0 0
Sep 15 04:31:42 smtp2 kernel: Node 0 Normal free:0kB min:0kB low:0kB
high:0kB active:0kB inactive:0kB present:0kB pages_scanned:0
all_unreclaimable? no
Sep 15 04:31:42 smtp2 kernel: lowmem_reserve[]: 0 0 0 0
Sep 15 04:31:42 smtp2 kernel: Node 0 HighMem free:0kB min:128kB
low:128kB high:128kB active:0kB inactive:0kB present:0kB pages_scanned:0
all_unreclaimable? no
Sep 15 04:31:42 smtp2 kernel: lowmem_reserve[]: 0 0 0 0
Sep 15 04:31:42 smtp2 kernel: Node 0 DMA: 2*4kB 4*8kB 2*16kB 5*32kB
4*64kB 1*128kB 2*256kB 0*512kB 1*1024kB 0*2048kB 0*4096kB = 2152kB
Sep 15 04:31:42 smtp2 kernel: Node 0 DMA32: 130*4kB 4*8kB 2*16kB 14*32kB
5*64kB 1*128kB 1*256kB 0*512kB 0*1024kB 0*2048kB 1*4096kB = 5832kB
Sep 15 04:31:43 smtp2 kernel: Node 0 Normal: empty
Sep 15 04:31:43 smtp2 kernel: Node 0 HighMem: empty
Sep 15 04:31:43 smtp2 kernel: 6467 pagecache pages
Sep 15 04:31:43 smtp2 kernel: Swap cache: add 613470, delete 609770,
find 99957/110522, race 3+8
Sep 15 04:31:43 smtp2 kernel: Free swap  = 0kB
Sep 15 04:31:43 smtp2 kernel: Total swap = 2096472kB
Sep 15 04:31:43 smtp2 kernel: Free swap:            0kB
Sep 15 04:31:43 smtp2 kernel: 524000 pages of RAM
Sep 15 04:31:43 smtp2 kernel: 9347 reserved pages
Sep 15 04:31:43 smtp2 kernel: 1067643 pages shared
Sep 15 04:31:43 smtp2 kernel: 3700 pages swap cached
Sep 15 04:31:43 smtp2 kernel: postdrop invoked oom-killer:
gfp_mask=0x201d2, order=0, oomkilladj=0
Sep 15 04:31:43 smtp2 kernel:
Sep 15 04:31:43 smtp2 kernel: Call Trace:
Sep 15 04:31:43 smtp2 kernel:  [<ffffffff800c3bda>] out_of_memory+0x8e/0x2f3
Sep 15 04:31:43 smtp2 kernel:  [<ffffffff8000f2ea>]
__alloc_pages+0x245/0x2ce
Sep 15 04:31:43 smtp2 kernel:  [<ffffffff80012a77>]
__do_page_cache_readahead+0xa2/0x1e6
Sep 15 04:31:43 smtp2 kernel:  [<ffffffff80054ab8>] sock_sendmsg+0xf3/0x110
Sep 15 04:31:43 smtp2 kernel:  [<ffffffff8000e87a>] link_path_walk+0xd3/0xe5
Sep 15 04:31:43 smtp2 kernel:  [<ffffffff80013406>]
filemap_nopage+0x148/0x322
Sep 15 04:31:43 smtp2 kernel:  [<ffffffff8000886a>]
__handle_mm_fault+0x1f8/0xe5c
Sep 15 04:31:43 smtp2 kernel:  [<ffffffff80066b9f>]
do_page_fault+0x4cb/0x830
Sep 15 04:31:43 smtp2 kernel:  [<ffffffff80059fdf>] hrtimer_cancel+0xc/0x16
Sep 15 04:31:43 smtp2 kernel:  [<ffffffff80063db7>] do_nanosleep+0x47/0x70
Sep 15 04:31:43 smtp2 kernel:  [<ffffffff80059ecc>]
hrtimer_nanosleep+0x58/0x118
Sep 15 04:31:43 smtp2 kernel:  [<ffffffff8005dde9>] error_exit+0x0/0x84
Sep 15 04:31:43 smtp2 kernel:
Sep 15 04:31:43 smtp2 kernel: Mem-info:
Sep 15 04:31:43 smtp2 kernel: Node 0 DMA per-cpu:
Sep 15 04:31:43 smtp2 kernel: cpu 0 hot: high 0, batch 1 used:0
Sep 15 04:31:43 smtp2 kernel: cpu 0 cold: high 0, batch 1 used:0
Sep 15 04:31:43 smtp2 kernel: cpu 1 hot: high 0, batch 1 used:0
Sep 15 04:31:43 smtp2 kernel: cpu 1 cold: high 0, batch 1 used:0
Sep 15 04:31:43 smtp2 kernel: Node 0 DMA32 per-cpu:
Sep 15 04:31:43 smtp2 kernel: cpu 0 hot: high 186, batch 31 used:113
Sep 15 04:31:43 smtp2 kernel: cpu 0 cold: high 62, batch 15 used:32
Sep 15 04:31:43 smtp2 kernel: cpu 1 hot: high 186, batch 31 used:171
Sep 15 04:31:43 smtp2 kernel: cpu 1 cold: high 62, batch 15 used:57
Sep 15 04:31:43 smtp2 kernel: Node 0 Normal per-cpu: empty
Sep 15 04:31:43 smtp2 kernel: Node 0 HighMem per-cpu: empty
Sep 15 04:31:43 smtp2 kernel: Free pages:        8388kB (0kB HighMem)
Sep 15 04:31:43 smtp2 kernel: Active:180176 inactive:164113 dirty:0
writeback:0 unstable:0 free:2097 slab:32458 mapped-file:2711
mapped-anon:352991 pagetables:97276
Sep 15 04:31:43 smtp2 kernel: Node 0 DMA free:2152kB min:28kB low:32kB
high:40kB active:0kB inactive:0kB present:10716kB pages_scanned:0
all_unreclaimable? yes
Sep 15 04:31:43 smtp2 kernel: lowmem_reserve[]: 0 2003 2003 2003
Sep 15 04:31:43 smtp2 kernel: Node 0 DMA32 free:6236kB min:5708kB
low:7132kB high:8560kB active:719796kB inactive:657360kB
present:2051184kB pages_scanned:531325 all_unreclaimable? no
Sep 15 04:31:43 smtp2 kernel: lowmem_reserve[]: 0 0 0 0
Sep 15 04:31:43 smtp2 kernel: Node 0 Normal free:0kB min:0kB low:0kB
high:0kB active:0kB inactive:0kB present:0kB pages_scanned:0
all_unreclaimable? no
Sep 15 04:31:43 smtp2 kernel: lowmem_reserve[]: 0 0 0 0
Sep 15 04:31:43 smtp2 kernel: Node 0 HighMem free:0kB min:128kB
low:128kB high:128kB active:0kB inactive:0kB present:0kB pages_scanned:0
all_unreclaimable? no
Sep 15 04:31:43 smtp2 kernel: lowmem_reserve[]: 0 0 0 0
Sep 15 04:31:43 smtp2 kernel: Node 0 DMA: 2*4kB 4*8kB 2*16kB 5*32kB
4*64kB 1*128kB 2*256kB 0*512kB 1*1024kB 0*2048kB 0*4096kB = 2152kB
Sep 15 04:31:43 smtp2 kernel: Node 0 DMA32: 235*4kB 2*8kB 2*16kB 14*32kB
5*64kB 1*128kB 1*256kB 0*512kB 0*1024kB 0*2048kB 1*4096kB = 6236kB
Sep 15 04:31:43 smtp2 kernel: Node 0 Normal: empty
Sep 15 04:31:43 smtp2 kernel: Node 0 HighMem: empty
Sep 15 04:31:43 smtp2 kernel: 6501 pagecache pages
Sep 15 04:31:43 smtp2 kernel: Swap cache: add 613655, delete 609955,
find 99957/110524, race 3+8
Sep 15 04:31:44 smtp2 kernel: Free swap  = 0kB
Sep 15 04:31:44 smtp2 kernel: Total swap = 2096472kB
Sep 15 04:31:45 smtp2 kernel: Free swap:            0kB
Sep 15 04:31:45 smtp2 kernel: 524000 pages of RAM
Sep 15 04:31:45 smtp2 kernel: 9347 reserved pages
Sep 15 04:31:46 smtp2 kernel: 1066858 pages shared
Sep 15 04:31:46 smtp2 kernel: 3700 pages swap cached
Sep 15 04:31:46 smtp2 kernel: Out of memory: Killed process 3268 (crond).
Sep 15 04:31:46 smtp2 kernel: block drbd0: helper command: /sbin/drbdadm
after-resync-target minor-0 exit code 0 (0x0)
Sep 15 04:31:46 smtp2 kernel: block drbd0: md_sync_timer expired! Worker
calls drbd_md_sync().
Sep 15 04:31:46 smtp2 kernel: modclusterd invoked oom-killer:
gfp_mask=0x201d2, order=0, oomkilladj=0
Sep 15 04:31:46 smtp2 kernel:
Sep 15 04:31:46 smtp2 kernel: Call Trace:
Sep 15 04:31:46 smtp2 kernel:  [<ffffffff800c3bda>] out_of_memory+0x8e/0x2f3
Sep 15 04:31:46 smtp2 kernel:  [<ffffffff8000f2ea>]
__alloc_pages+0x245/0x2ce
Sep 15 04:31:46 smtp2 kernel:  [<ffffffff80012a77>]
__do_page_cache_readahead+0xa2/0x1e6
Sep 15 04:31:46 smtp2 kernel:  [<ffffffff80063a05>]
__wait_on_bit_lock+0x5b/0x66
Sep 15 04:31:46 smtp2 kernel:  [<ffffffff8003f626>] __lock_page+0x5e/0x64
Sep 15 04:31:46 smtp2 kernel:  [<ffffffff80013406>]
filemap_nopage+0x148/0x322
Sep 15 04:31:46 smtp2 kernel:  [<ffffffff8000886a>]
__handle_mm_fault+0x1f8/0xe5c
Sep 15 04:31:46 smtp2 kernel:  [<ffffffff80066b9f>]
do_page_fault+0x4cb/0x830
Sep 15 04:31:46 smtp2 kernel:  [<ffffffff800320b8>] sys_faccessat+0xe4/0x18d
Sep 15 04:31:46 smtp2 kernel:  [<ffffffff800b4629>]
audit_syscall_entry+0x14a/0x1b3
Sep 15 04:31:46 smtp2 kernel:  [<ffffffff8005dde9>] error_exit+0x0/0x84
Sep 15 04:31:46 smtp2 kernel:
Sep 15 04:31:46 smtp2 kernel: Mem-info:
Sep 15 04:31:47 smtp2 kernel: Node 0 DMA per-cpu:
Sep 15 04:31:47 smtp2 kernel: cpu 0 hot: high 0, batch 1 used:0
Sep 15 04:31:47 smtp2 kernel: cpu 0 cold: high 0, batch 1 used:0
Sep 15 04:31:47 smtp2 kernel: cpu 1 hot: high 0, batch 1 used:0
Sep 15 04:31:47 smtp2 kernel: cpu 1 cold: high 0, batch 1 used:0
Sep 15 04:31:47 smtp2 kernel: Node 0 DMA32 per-cpu:
Sep 15 04:31:47 smtp2 kernel: cpu 0 hot: high 186, batch 31 used:30
Sep 15 04:31:47 smtp2 kernel: cpu 0 cold: high 62, batch 15 used:14
Sep 15 04:31:47 smtp2 kernel: cpu 1 hot: high 186, batch 31 used:6
Sep 15 04:31:47 smtp2 kernel: cpu 1 cold: high 62, batch 15 used:31
Sep 15 04:31:47 smtp2 kernel: Node 0 Normal per-cpu: empty
Sep 15 04:31:47 smtp2 kernel: Node 0 HighMem per-cpu: empty
Sep 15 04:31:47 smtp2 kernel: Free pages:        7804kB (0kB HighMem)
Sep 15 04:31:47 smtp2 kernel: Active:223476 inactive:131748 dirty:0
writeback:0 unstable:0 free:1951 slab:32653 mapped-file:2757
mapped-anon:353073 pagetables:97208
Sep 15 04:31:47 smtp2 kernel: Node 0 DMA free:2152kB min:28kB low:32kB
high:40kB active:0kB inactive:0kB present:10716kB pages_scanned:0
all_unreclaimable? yes
Sep 15 04:31:47 smtp2 kernel: lowmem_reserve[]: 0 2003 2003 2003
Sep 15 04:31:47 smtp2 kernel: Node 0 DMA32 free:5652kB min:5708kB
low:7132kB high:8560kB active:895736kB inactive:525032kB
present:2051184kB pages_scanned:3253851 all_unreclaimable? yes
Sep 15 04:31:47 smtp2 kernel: lowmem_reserve[]: 0 0 0 0
Sep 15 04:31:47 smtp2 kernel: Node 0 Normal free:0kB min:0kB low:0kB
high:0kB active:0kB inactive:0kB present:0kB pages_scanned:0
all_unreclaimable? no
Sep 15 04:31:47 smtp2 kernel: lowmem_reserve[]: 0 0 0 0
Sep 15 04:31:47 smtp2 kernel: Node 0 HighMem free:0kB min:128kB
low:128kB high:128kB active:0kB inactive:0kB present:0kB pages_scanned:0
all_unreclaimable? no
Sep 15 04:31:47 smtp2 kernel: lowmem_reserve[]: 0 0 0 0
Sep 15 04:31:47 smtp2 kernel: Node 0 DMA: 2*4kB 4*8kB 2*16kB 5*32kB
4*64kB 1*128kB 2*256kB 0*512kB 1*1024kB 0*2048kB 0*4096kB = 2152kB
Sep 15 04:31:47 smtp2 kernel: Node 0 DMA32: 19*4kB 27*8kB 3*16kB 12*32kB
5*64kB 2*128kB 1*256kB 0*512kB 0*1024kB 0*2048kB 1*4096kB = 5652kB
Sep 15 04:31:47 smtp2 kernel: Node 0 Normal: empty
Sep 15 04:31:47 smtp2 kernel: Node 0 HighMem: empty
Sep 15 04:31:47 smtp2 kernel: 6741 pagecache pages
Sep 15 04:31:47 smtp2 kernel: Swap cache: add 616939, delete 613225,
find 100005/110937, race 3+9
Sep 15 04:31:47 smtp2 kernel: Free swap  = 0kB
Sep 15 04:31:47 smtp2 kernel: Total swap = 2096472kB
Sep 15 04:31:47 smtp2 kernel: Free swap:            0kB
Sep 15 04:31:47 smtp2 kernel: 524000 pages of RAM
Sep 15 04:31:47 smtp2 kernel: 9347 reserved pages
Sep 15 04:31:47 smtp2 kernel: 1057504 pages shared
Sep 15 04:31:47 smtp2 kernel: 3714 pages swap cached
Sep 15 04:31:47 smtp2 kernel: Out of memory: Killed process 3284 (crond).
Sep 15 04:31:47 smtp2 kernel: modclusterd invoked oom-killer:
gfp_mask=0x201d2, order=0, oomkilladj=0
Sep 15 04:31:47 smtp2 kernel:
Sep 15 04:31:47 smtp2 kernel: Call Trace:
Sep 15 04:31:47 smtp2 kernel:  [<ffffffff800c3bda>] out_of_memory+0x8e/0x2f3
Sep 15 04:31:47 smtp2 kernel:  [<ffffffff8000f2ea>]
__alloc_pages+0x245/0x2ce
Sep 15 04:31:47 smtp2 kernel:  [<ffffffff80012a77>]
__do_page_cache_readahead+0xa2/0x1e6
Sep 15 04:31:47 smtp2 kernel:  [<ffffffff80063a05>]
__wait_on_bit_lock+0x5b/0x66
Sep 15 04:31:47 smtp2 kernel:  [<ffffffff8003f626>] __lock_page+0x5e/0x64
Sep 15 04:31:47 smtp2 kernel:  [<ffffffff80013406>]
filemap_nopage+0x148/0x322
Sep 15 04:31:52 smtp2 kernel:  [<ffffffff8000886a>]
__handle_mm_fault+0x1f8/0xe5c
Sep 15 04:31:52 smtp2 kernel:  [<ffffffff80066b9f>]
do_page_fault+0x4cb/0x830
Sep 15 04:31:52 smtp2 kernel:  [<ffffffff800320b8>] sys_faccessat+0xe4/0x18d
Sep 15 04:31:52 smtp2 kernel:  [<ffffffff800b4629>]
audit_syscall_entry+0x14a/0x1b3
Sep 15 04:31:52 smtp2 kernel:  [<ffffffff8005dde9>] error_exit+0x0/0x84
Sep 15 04:31:52 smtp2 kernel:
Sep 15 04:31:52 smtp2 kernel: Mem-info:
Sep 15 04:31:52 smtp2 kernel: Node 0 DMA per-cpu:
Sep 15 04:31:52 smtp2 kernel: cpu 0 hot: high 0, batch 1 used:0
Sep 15 04:31:52 smtp2 kernel: cpu 0 cold: high 0, batch 1 used:0
Sep 15 04:31:52 smtp2 kernel: cpu 1 hot: high 0, batch 1 used:0
Sep 15 04:31:52 smtp2 kernel: cpu 1 cold: high 0, batch 1 used:0
Sep 15 04:31:52 smtp2 kernel: Node 0 DMA32 per-cpu:
Sep 15 04:31:54 smtp2 kernel: cpu 0 hot: high 186, batch 31 used:0
Sep 15 04:31:54 smtp2 kernel: cpu 0 cold: high 62, batch 15 used:14
Sep 15 04:31:54 smtp2 kernel: cpu 1 hot: high 186, batch 31 used:58
Sep 15 04:31:54 smtp2 kernel: cpu 1 cold: high 62, batch 15 used:26
Sep 15 04:31:54 smtp2 kernel: Node 0 Normal per-cpu: empty
Sep 15 04:31:54 smtp2 kernel: Node 0 HighMem per-cpu: empty
Sep 15 04:31:54 smtp2 kernel: Free pages:        7812kB (0kB HighMem)
Sep 15 04:31:54 smtp2 kernel: Active:114250 inactive:241428 dirty:0
writeback:1 unstable:0 free:1953 slab:32655 mapped-file:2751
mapped-anon:353072 pagetables:97208
Sep 15 04:31:54 smtp2 kernel: Node 0 DMA free:2152kB min:28kB low:32kB
high:40kB active:0kB inactive:0kB present:10716kB pages_scanned:0
all_unreclaimable? yes
Sep 15 04:31:54 smtp2 kernel: lowmem_reserve[]: 0 2003 2003 2003
Sep 15 04:31:54 smtp2 kernel: Node 0 DMA32 free:5660kB min:5708kB
low:7132kB high:8560kB active:457000kB inactive:965712kB
present:2051184kB pages_scanned:4264628 all_unreclaimable? yes
Sep 15 04:31:54 smtp2 kernel: lowmem_reserve[]: 0 0 0 0
Sep 15 04:31:55 smtp2 kernel: Node 0 Normal free:0kB min:0kB low:0kB
high:0kB active:0kB inactive:0kB present:0kB pages_scanned:0
all_unreclaimable? no
Sep 15 04:31:55 smtp2 kernel: lowmem_reserve[]: 0 0 0 0
Sep 15 04:31:55 smtp2 kernel: Node 0 HighMem free:0kB min:128kB
low:128kB high:128kB active:0kB inactive:0kB present:0kB pages_scanned:0
all_unreclaimable? no
Sep 15 04:31:55 smtp2 kernel: lowmem_reserve[]: 0 0 0 0
Sep 15 04:31:55 smtp2 kernel: Node 0 DMA: 2*4kB 4*8kB 2*16kB 5*32kB
4*64kB 1*128kB 2*256kB 0*512kB 1*1024kB 0*2048kB 0*4096kB = 2152kB
Sep 15 04:31:55 smtp2 kernel: Node 0 DMA32: 17*4kB 29*8kB 3*16kB 12*32kB
5*64kB 2*128kB 1*256kB 0*512kB 0*1024kB 0*2048kB 1*4096kB = 5660kB
Sep 15 04:31:55 smtp2 kernel: Node 0 Normal: empty
Sep 15 04:31:55 smtp2 kernel: Node 0 HighMem: empty
Sep 15 04:31:55 smtp2 kernel: 6705 pagecache pages
Sep 15 04:31:55 smtp2 kernel: Swap cache: add 616966, delete 613253,
find 100005/110937, race 3+9
Sep 15 04:31:55 smtp2 kernel: Free swap  = 0kB
Sep 15 04:31:55 smtp2 kernel: Total swap = 2096472kB
Sep 15 04:31:55 smtp2 kernel: Free swap:            0kB
Sep 15 04:31:55 smtp2 kernel: 524000 pages of RAM
Sep 15 04:31:55 smtp2 kernel: 9347 reserved pages
Sep 15 04:31:55 smtp2 kernel: 1055243 pages shared
Sep 15 04:31:55 smtp2 kernel: 3713 pages swap cached
Sep 15 04:31:59 smtp2 nss-mysql[24311]: _nss_mysql_db_connect:
connection failed: Lost connection to MySQL server at 'reading
authorization packet', system error: 0
Sep 15 04:31:59 smtp2 nss-mysql[24311]: setent(1): connection to the
database failed.
Sep 15 04:35:20 smtp2 kernel: modclusterd invoked oom-killer:
gfp_mask=0x201d2, order=0, oomkilladj=0
Sep 15 04:35:27 smtp2 kernel:
Sep 15 04:35:27 smtp2 kernel: Call Trace:
Sep 15 04:35:29 smtp2 kernel:  [<ffffffff800c3bda>] out_of_memory+0x8e/0x2f3
Sep 15 04:35:29 smtp2 kernel:  [<ffffffff8000f2ea>]
__alloc_pages+0x245/0x2ce
Sep 15 04:35:29 smtp2 kernel:  [<ffffffff80012a77>]
__do_page_cache_readahead+0xa2/0x1e6
Sep 15 04:35:29 smtp2 kernel:  [<ffffffff80063a05>]
__wait_on_bit_lock+0x5b/0x66
Sep 15 04:35:29 smtp2 kernel:  [<ffffffff8003f626>] __lock_page+0x5e/0x64
Sep 15 04:35:29 smtp2 kernel:  [<ffffffff80013406>]
filemap_nopage+0x148/0x322
Sep 15 04:35:29 smtp2 kernel:  [<ffffffff8000886a>]
__handle_mm_fault+0x1f8/0xe5c
Sep 15 04:35:29 smtp2 kernel:  [<ffffffff80066b9f>]
do_page_fault+0x4cb/0x830
Sep 15 04:35:29 smtp2 kernel:  [<ffffffff800320b8>] sys_faccessat+0xe4/0x18d
Sep 15 04:35:29 smtp2 kernel:  [<ffffffff800b4629>]
audit_syscall_entry+0x14a/0x1b3
Sep 15 04:35:29 smtp2 kernel:  [<ffffffff8005dde9>] error_exit+0x0/0x84
Sep 15 04:35:29 smtp2 kernel:
Sep 15 04:35:30 smtp2 kernel: Mem-info:
Sep 15 04:35:31 smtp2 snmpd[2489]: Connection from UDP:
[206.16.46.12]:59774
Sep 15 04:35:31 smtp2 kernel: Node 0 DMA per-cpu:
Sep 15 04:35:31 smtp2 snmpd[2489]: Received SNMP packet(s) from UDP:
[206.16.46.12]:59774
Sep 15 04:35:31 smtp2 kernel: cpu 0 hot: high 0, batch 1 used:0
Sep 15 04:35:32 smtp2 snmpd[2489]: Connection from UDP:
[206.16.46.12]:59774
Sep 15 04:35:32 smtp2 kernel: cpu 0 cold: high 0, batch 1 used:0
Sep 15 04:35:32 smtp2 snmpd[2489]: Connection from UDP:
[206.16.46.12]:59774
Sep 15 04:35:32 smtp2 kernel: cpu 1 hot: high 0, batch 1 used:0
Sep 15 04:35:32 smtp2 snmpd[2489]: Connection from UDP:
[206.16.46.12]:59774
Sep 15 04:35:32 smtp2 kernel: cpu 1 cold: high 0, batch 1 used:0
Sep 15 04:35:32 smtp2 snmpd[2489]: Connection from UDP:
[206.16.46.12]:59774
Sep 15 04:35:32 smtp2 kernel: Node 0 DMA32 per-cpu:
Sep 15 04:35:32 smtp2 snmpd[2489]: Connection from UDP:
[206.16.46.12]:59774
Sep 15 04:35:32 smtp2 kernel: cpu 0 hot: high 186, batch 31 used:30
Sep 15 04:35:32 smtp2 snmpd[2489]: Connection from UDP:
[206.16.46.12]:59774
Sep 15 04:35:32 smtp2 kernel: cpu 0 cold: high 62, batch 15 used:14
Sep 15 04:35:32 smtp2 snmpd[2489]: Connection from UDP:
[206.16.46.12]:59774
Sep 15 04:35:32 smtp2 kernel: cpu 1 hot: high 186, batch 31 used:44
Sep 15 04:35:32 smtp2 snmpd[2489]: Connection from UDP:
[206.16.46.12]:59774
Sep 15 04:35:32 smtp2 kernel: cpu 1 cold: high 62, batch 15 used:58
Sep 15 04:35:32 smtp2 snmpd[2489]: Connection from UDP:
[206.16.46.12]:59774
Sep 15 04:35:32 smtp2 kernel: Node 0 Normal per-cpu: empty
Sep 15 04:35:32 smtp2 snmpd[2489]: Connection from UDP:
[206.16.46.12]:59774
Sep 15 04:35:32 smtp2 kernel: Node 0 HighMem per-cpu: empty
Sep 15 04:35:32 smtp2 snmpd[2489]: Connection from UDP:
[206.16.46.12]:59774
Sep 15 04:35:32 smtp2 kernel: Free pages:        7824kB (0kB HighMem)
Sep 15 04:35:32 smtp2 snmpd[2489]: Connection from UDP:
[206.16.46.12]:59774
Sep 15 04:35:32 smtp2 kernel: Active:174258 inactive:180606 dirty:0
writeback:0 unstable:0 free:1956 slab:32676 mapped-file:2723
mapped-anon:353207 pagetables:97216
Sep 15 04:35:32 smtp2 snmpd[2489]: Connection from UDP:
[206.16.46.12]:59774
Sep 15 04:35:32 smtp2 kernel: Node 0 DMA free:2152kB min:28kB low:32kB
high:40kB active:0kB inactive:0kB present:10716kB pages_scanned:0
all_unreclaimable? yes
Sep 15 04:35:32 smtp2 snmpd[2489]: Connection from UDP:
[206.16.46.12]:59774
Sep 15 04:35:32 smtp2 kernel: lowmem_reserve[]: 0 2003 2003 2003
Sep 15 04:35:32 smtp2 snmpd[2489]: Connection from UDP:
[206.16.46.12]:59774
Sep 15 04:35:33 smtp2 kernel: Node 0 DMA32 free:5672kB min:5708kB
low:7132kB high:8560kB active:697032kB inactive:722424kB
present:2051184kB pages_scanned:2156508 all_unreclaimable? yes
Sep 15 04:35:33 smtp2 snmpd[2489]: Connection from UDP:
[206.16.46.12]:59774
Sep 15 04:35:33 smtp2 kernel: lowmem_reserve[]: 0 0 0 0
Sep 15 04:35:33 smtp2 snmpd[2489]: Connection from UDP:
[206.16.46.12]:59774
Sep 15 04:35:33 smtp2 kernel: Node 0 Normal free:0kB min:0kB low:0kB
high:0kB active:0kB inactive:0kB present:0kB pages_scanned:0
all_unreclaimable? no
Sep 15 04:35:33 smtp2 snmpd[2489]: Connection from UDP:
[206.16.46.12]:59774
Sep 15 04:35:33 smtp2 kernel: lowmem_reserve[]: 0 0 0 0
Sep 15 04:35:33 smtp2 snmpd[2489]: Connection from UDP:
[206.16.46.12]:59774
Sep 15 04:35:33 smtp2 kernel: Node 0 HighMem free:0kB min:128kB
low:128kB high:128kB active:0kB inactive:0kB present:0kB pages_scanned:0
all_unreclaimable? no
Sep 15 04:35:33 smtp2 snmpd[2489]: Connection from UDP:
[206.16.46.12]:59774
Sep 15 04:35:33 smtp2 kernel: lowmem_reserve[]: 0 0 0 0
Sep 15 04:35:33 smtp2 snmpd[2489]: Connection from UDP:
[206.16.46.12]:59774
Sep 15 04:35:33 smtp2 kernel: Node 0 DMA: 2*4kB 4*8kB 2*16kB 5*32kB
4*64kB 1*128kB 2*256kB 0*512kB 1*1024kB 0*2048kB 0*4096kB = 2152kB
Sep 15 04:35:33 smtp2 snmpd[2489]: Connection from UDP:
[206.16.46.12]:59774
Sep 15 04:35:33 smtp2 kernel: Node 0 DMA32: 12*4kB 33*8kB 3*16kB 12*32kB
5*64kB 2*128kB 1*256kB 0*512kB 0*1024kB 0*2048kB 1*4096kB = 5672kB
Sep 15 04:35:33 smtp2 snmpd[2489]: Connection from UDP:
[206.16.46.12]:59774
Sep 15 04:35:33 smtp2 kernel: Node 0 Normal: empty
Sep 15 04:35:33 smtp2 snmpd[2489]: Connection from UDP:
[206.16.46.12]:59774
Sep 15 04:35:33 smtp2 kernel: Node 0 HighMem: empty
Sep 15 04:35:33 smtp2 snmpd[2489]: Connection from UDP:
[206.16.46.12]:59774
Sep 15 04:35:33 smtp2 kernel: 6490 pagecache pages
Sep 15 04:35:33 smtp2 snmpd[2489]: Connection from UDP:
[206.16.46.12]:59774
Sep 15 04:35:33 smtp2 kernel: Swap cache: add 618956, delete 615255,
find 100105/111249, race 3+10
Sep 15 04:35:33 smtp2 snmpd[2489]: Connection from UDP:
[206.16.46.12]:59774
Sep 15 04:35:33 smtp2 kernel: Free swap  = 0kB
Sep 15 04:35:33 smtp2 snmpd[2489]: Connection from UDP:
[206.16.46.12]:59774
Sep 15 04:35:33 smtp2 kernel: Total swap = 2096472kB
Sep 15 04:35:33 smtp2 snmpd[2489]: Connection from UDP:
[206.16.46.12]:59774
Sep 15 04:35:33 smtp2 kernel: Free swap:            0kB
Sep 15 04:35:33 smtp2 snmpd[2489]: Connection from UDP:
[206.16.46.12]:59774
Sep 15 04:35:33 smtp2 kernel: 524000 pages of RAM
Sep 15 04:35:33 smtp2 snmpd[2489]: Connection from UDP:
[206.16.46.12]:59774
Sep 15 04:35:33 smtp2 kernel: 9347 reserved pages
Sep 15 04:35:33 smtp2 snmpd[2489]: Connection from UDP:
[206.16.46.12]:59774
Sep 15 04:35:33 smtp2 kernel: 1055994 pages shared
Sep 15 04:35:33 smtp2 snmpd[2489]: Connection from UDP:
[206.16.46.12]:59774
Sep 15 04:35:33 smtp2 kernel: 3701 pages swap cached
Sep 15 04:35:33 smtp2 snmpd[2489]: Connection from UDP:
[206.16.46.12]:59774
Sep 15 04:35:33 smtp2 kernel: Out of memory: Killed process 3301 (crond).
Sep 15 04:35:34 smtp2 snmpd[2489]: Connection from UDP:
[206.16.46.12]:59774
Sep 15 04:35:35 smtp2 kernel: postdrop invoked oom-killer:
gfp_mask=0x201d2, order=0, oomkilladj=0
Sep 15 04:35:36 smtp2 snmpd[2489]: Connection from UDP:
[206.16.46.12]:59774
Sep 15 04:35:36 smtp2 kernel:
Sep 15 04:36:01 smtp2 snmpd[2489]: Connection from UDP:
[206.16.46.12]:59774
Sep 15 04:36:07 smtp2 kernel: Call Trace:
Sep 15 04:36:15 smtp2 kernel:  [<ffffffff800c3bda>] out_of_memory+0x8e/0x2f3
Sep 15 04:36:15 smtp2 kernel:  [<ffffffff8000f2ea>]
__alloc_pages+0x245/0x2ce
Sep 15 04:36:15 smtp2 kernel:  [<ffffffff80012a77>]
__do_page_cache_readahead+0xa2/0x1e6
Sep 15 04:36:15 smtp2 kernel:  [<ffffffff80063a05>]
__wait_on_bit_lock+0x5b/0x66
Sep 15 04:36:15 smtp2 kernel:  [<ffffffff8003f626>] __lock_page+0x5e/0x64
Sep 15 04:36:15 smtp2 kernel:  [<ffffffff80013406>]
filemap_nopage+0x148/0x322
Sep 15 04:36:15 smtp2 kernel:  [<ffffffff8000886a>]
__handle_mm_fault+0x1f8/0xe5c
Sep 15 04:36:15 smtp2 kernel:  [<ffffffff80066b9f>]
do_page_fault+0x4cb/0x830
Sep 15 04:36:15 smtp2 kernel:  [<ffffffff80059fdf>] hrtimer_cancel+0xc/0x16
Sep 15 04:36:15 smtp2 kernel:  [<ffffffff80063db7>] do_nanosleep+0x47/0x70
Sep 15 04:36:15 smtp2 kernel:  [<ffffffff80059ecc>]
hrtimer_nanosleep+0x58/0x118
Sep 15 04:36:15 smtp2 kernel:  [<ffffffff8005dde9>] error_exit+0x0/0x84
Sep 15 04:36:15 smtp2 kernel:
Sep 15 04:36:16 smtp2 kernel: Mem-info:
Sep 15 04:36:16 smtp2 kernel: Node 0 DMA per-cpu:
Sep 15 04:36:16 smtp2 kernel: cpu 0 hot: high 0, batch 1 used:0
Sep 15 04:36:16 smtp2 kernel: cpu 0 cold: high 0, batch 1 used:0
Sep 15 04:36:16 smtp2 kernel: cpu 1 hot: high 0, batch 1 used:0
Sep 15 04:36:16 smtp2 kernel: cpu 1 cold: high 0, batch 1 used:0
Sep 15 04:36:16 smtp2 kernel: Node 0 DMA32 per-cpu:
Sep 15 04:36:16 smtp2 kernel: cpu 0 hot: high 186, batch 31 used:30
Sep 15 04:36:16 smtp2 kernel: cpu 0 cold: high 62, batch 15 used:45
Sep 15 04:36:16 smtp2 kernel: cpu 1 hot: high 186, batch 31 used:48
Sep 15 04:36:16 smtp2 kernel: cpu 1 cold: high 62, batch 15 used:58
Sep 15 04:36:16 smtp2 kernel: Node 0 Normal per-cpu: empty
Sep 15 04:36:16 smtp2 kernel: Node 0 HighMem per-cpu: empty
Sep 15 04:36:16 smtp2 kernel: Free pages:        7824kB (0kB HighMem)
Sep 15 04:36:16 smtp2 kernel: Active:223642 inactive:131077 dirty:0
writeback:0 unstable:0 free:1956 slab:32672 mapped-file:2722
mapped-anon:353207 pagetables:97216
Sep 15 04:36:16 smtp2 snmpd[2489]: Connection from UDP:
[206.16.46.12]:59774
Sep 15 04:36:16 smtp2 kernel: Node 0 DMA free:2152kB min:28kB low:32kB
high:40kB active:0kB inactive:0kB present:10716kB pages_scanned:0
all_unreclaimable? yes
Sep 15 04:36:16 smtp2 snmpd[2489]: Connection from UDP:
[206.16.46.12]:59774
Sep 15 04:36:16 smtp2 kernel: lowmem_reserve[]: 0 2003 2003 2003
Sep 15 04:36:16 smtp2 snmpd[2489]: Connection from UDP:
[206.16.46.12]:59774
Sep 15 04:36:16 smtp2 kernel: Node 0 DMA32 free:5672kB min:5708kB
low:7132kB high:8560kB active:894576kB inactive:524428kB
present:2051184kB pages_scanned:5377722 all_unreclaimable? yes
Sep 15 04:36:16 smtp2 snmpd[2489]: Connection from UDP:
[206.16.46.12]:59774
Sep 15 04:36:16 smtp2 kernel: lowmem_reserve[]: 0 0 0 0
Sep 15 04:36:16 smtp2 snmpd[2489]: Connection from UDP:
[206.16.46.12]:59774
Sep 15 04:36:16 smtp2 kernel: Node 0 Normal free:0kB min:0kB low:0kB
high:0kB active:0kB inactive:0kB present:0kB pages_scanned:0
all_unreclaimable? no
Sep 15 04:36:16 smtp2 kernel: lowmem_reserve[]: 0 0 0 0
Sep 15 04:36:16 smtp2 kernel: Node 0 HighMem free:0kB min:128kB
low:128kB high:128kB active:0kB inactive:0kB present:0kB pages_scanned:0
all_unreclaimable? no
Sep 15 04:36:16 smtp2 kernel: lowmem_reserve[]: 0 0 0 0
Sep 15 04:36:16 smtp2 snmpd[2489]: Connection from UDP:
[206.16.46.12]:59774
Sep 15 04:36:16 smtp2 kernel: Node 0 DMA: 2*4kB 4*8kB 2*16kB 5*32kB
4*64kB 1*128kB 2*256kB 0*512kB 1*1024kB 0*2048kB 0*4096kB = 2152kB
Sep 15 04:36:16 smtp2 snmpd[2489]: Connection from UDP:
[206.16.46.12]:59774
Sep 15 04:36:16 smtp2 kernel: Node 0 DMA32: 12*4kB 33*8kB 3*16kB 12*32kB
5*64kB 2*128kB 1*256kB 0*512kB 0*1024kB 0*2048kB 1*4096kB = 5672kB
Sep 15 04:36:16 smtp2 snmpd[2489]: Connection from UDP:
[206.16.46.12]:59774
Sep 15 04:36:16 smtp2 kernel: Node 0 Normal: empty
Sep 15 04:36:16 smtp2 snmpd[2489]: Connection from UDP:
[206.16.46.12]:59774
Sep 15 04:36:16 smtp2 kernel: Node 0 HighMem: empty
Sep 15 04:36:16 smtp2 snmpd[2489]: Connection from UDP:
[206.16.46.12]:59774
Sep 15 04:36:16 smtp2 kernel: 6459 pagecache pages
Sep 15 04:36:16 smtp2 snmpd[2489]: Connection from UDP:
[206.16.46.12]:59774
Sep 15 04:36:16 smtp2 kernel: Swap cache: add 618956, delete 615255,
find 100105/111249, race 3+10
Sep 15 04:36:16 smtp2 snmpd[2489]: Connection from UDP:
[206.16.46.12]:59774
Sep 15 04:36:16 smtp2 kernel: Free swap  = 0kB
Sep 15 04:36:16 smtp2 snmpd[2489]: Connection from UDP:
[206.16.46.12]:59774
Sep 15 04:36:16 smtp2 kernel: Total swap = 2096472kB
Sep 15 04:36:16 smtp2 snmpd[2489]: Connection from UDP:
[206.16.46.12]:59774
Sep 15 04:36:16 smtp2 kernel: Free swap:            0kB
Sep 15 04:36:16 smtp2 snmpd[2489]: Connection from UDP:
[206.16.46.12]:59774
Sep 15 04:36:16 smtp2 kernel: 524000 pages of RAM
Sep 15 04:36:16 smtp2 snmpd[2489]: Connection from UDP:
[206.16.46.12]:59774
Sep 15 04:36:16 smtp2 kernel: 9347 reserved pages
Sep 15 04:36:16 smtp2 snmpd[2489]: Connection from UDP:
[206.16.46.12]:59774
Sep 15 04:36:16 smtp2 kernel: 1056093 pages shared
Sep 15 04:36:16 smtp2 snmpd[2489]: Connection from UDP:
[206.16.46.12]:59774
Sep 15 04:36:16 smtp2 kernel: 3701 pages swap cached
Sep 15 04:36:16 smtp2 snmpd[2489]: Connection from UDP:
[206.16.46.12]:59774
Sep 15 04:36:16 smtp2 kernel: postdrop invoked oom-killer:
gfp_mask=0x4d0, order=0, oomkilladj=0
Sep 15 04:36:16 smtp2 snmpd[2489]: Connection from UDP:
[206.16.46.12]:59774
Sep 15 04:36:16 smtp2 kernel:
Sep 15 04:36:16 smtp2 snmpd[2489]: Connection from UDP:
[206.16.46.12]:59774
Sep 15 04:36:16 smtp2 kernel: Call Trace:
Sep 15 04:36:16 smtp2 snmpd[2489]: Connection from UDP:
[206.16.46.12]:59774
Sep 15 04:36:16 smtp2 kernel:  [<ffffffff800c3bda>] out_of_memory+0x8e/0x2f3
Sep 15 04:36:16 smtp2 snmpd[2489]: Connection from UDP:
[206.16.46.12]:59774
Sep 15 04:36:16 smtp2 kernel:  [<ffffffff8000f2ea>]
__alloc_pages+0x245/0x2ce
Sep 15 04:36:16 smtp2 snmpd[2489]: Connection from UDP:
[206.16.46.12]:59774
Sep 15 04:36:16 smtp2 kernel:  [<ffffffff8001748c>] cache_grow+0x133/0x381
Sep 15 04:36:16 smtp2 snmpd[2489]: Connection from UDP:
[206.16.46.12]:59774
Sep 15 04:36:16 smtp2 kernel:  [<ffffffff8005bc74>]
cache_alloc_refill+0x136/0x186
Sep 15 04:36:16 smtp2 snmpd[2489]: Connection from UDP:
[206.16.46.12]:59774
Sep 15 04:36:16 smtp2 kernel:  [<ffffffff800d6fbf>] __kmalloc+0x95/0x9f
Sep 15 04:36:16 smtp2 snmpd[2489]: Connection from UDP:
[206.16.46.12]:59774
Sep 15 04:36:16 smtp2 kernel:  [<ffffffff8002deee>] __alloc_skb+0x5c/0x122
Sep 15 04:36:16 smtp2 snmpd[2489]: Connection from UDP:
[206.16.46.12]:59774
Sep 15 04:36:16 smtp2 kernel:  [<ffffffff80216292>]
sock_alloc_send_skb+0x74/0x1dc
Sep 15 04:36:16 smtp2 snmpd[2489]: Connection from UDP:
[206.16.46.12]:59774
Sep 15 04:36:16 smtp2 kernel:  [<ffffffff80063098>] thread_return+0x62/0xfe
Sep 15 04:36:16 smtp2 snmpd[2489]: Connection from UDP:
[206.16.46.12]:59774
Sep 15 04:36:16 smtp2 kernel:  [<ffffffff80268e42>]
unix_dgram_sendmsg+0x17f/0x4e0
Sep 15 04:36:16 smtp2 snmpd[2489]: Connection from UDP:
[206.16.46.12]:59774
Sep 15 04:36:16 smtp2 kernel:  [<ffffffff80054ab8>] sock_sendmsg+0xf3/0x110
Sep 15 04:36:16 smtp2 snmpd[2489]: Connection from UDP:
[206.16.46.12]:59774
Sep 15 04:36:16 smtp2 kernel:  [<ffffffff8000e87a>] link_path_walk+0xd3/0xe5
Sep 15 04:36:16 smtp2 snmpd[2489]: Connection from UDP:
[206.16.46.12]:59774
Sep 15 04:36:16 smtp2 kernel:  [<ffffffff8009dbae>]
autoremove_wake_function+0x0/0x2e
Sep 15 04:36:16 smtp2 snmpd[2489]: Connection from UDP:
[206.16.46.12]:59774
Sep 15 04:36:16 smtp2 kernel:  [<ffffffff8000c394>]
_atomic_dec_and_lock+0x39/0x57
Sep 15 04:36:16 smtp2 snmpd[2489]: Connection from UDP:
[206.16.46.12]:59774
Sep 15 04:36:16 smtp2 kernel:  [<ffffffff8002c7f5>]
mntput_no_expire+0x19/0x89
Sep 15 04:36:16 smtp2 snmpd[2489]: Connection from UDP:
[206.16.46.12]:59774
Sep 15 04:36:16 smtp2 kernel:  [<ffffffff80214e4d>] sys_sendto+0x11c/0x14f
Sep 15 04:36:16 smtp2 snmpd[2489]: Connection from UDP:
[206.16.46.12]:59774
Sep 15 04:36:16 smtp2 kernel:  [<ffffffff8005d28d>] tracesys+0xd5/0xe0
Sep 15 04:36:17 smtp2 snmpd[2489]: Connection from UDP:
[206.16.46.12]:59774
Sep 15 04:36:17 smtp2 kernel:
Sep 15 04:36:17 smtp2 kernel: Mem-info:
Sep 15 04:36:17 smtp2 kernel: Node 0 DMA per-cpu:
Sep 15 04:36:17 smtp2 kernel: cpu 0 hot: high 0, batch 1 used:0
Sep 15 04:36:17 smtp2 kernel: cpu 0 cold: high 0, batch 1 used:0
Sep 15 04:36:18 smtp2 snmpd[2489]: Connection from UDP:
[206.16.46.12]:59774
Sep 15 04:36:18 smtp2 kernel: cpu 1 hot: high 0, batch 1 used:0
Sep 15 04:36:18 smtp2 snmpd[2489]: Connection from UDP:
[206.16.46.12]:59774
Sep 15 04:36:18 smtp2 kernel: cpu 1 cold: high 0, batch 1 used:0
Sep 15 04:36:18 smtp2 snmpd[2489]: Connection from UDP:
[206.16.46.12]:59774
Sep 15 04:36:18 smtp2 kernel: Node 0 DMA32 per-cpu:
Sep 15 04:36:18 smtp2 snmpd[2489]: Connection from UDP:
[206.16.46.12]:59774
Sep 15 04:36:18 smtp2 kernel: cpu 0 hot: high 186, batch 31 used:69
Sep 15 04:36:18 smtp2 snmpd[2489]: Connection from UDP:
[206.16.46.12]:59774
Sep 15 04:36:18 smtp2 kernel: cpu 0 cold: high 62, batch 15 used:0
Sep 15 04:36:18 smtp2 snmpd[2489]: Connection from UDP:
[206.16.46.12]:59774
Sep 15 04:36:18 smtp2 kernel: cpu 1 hot: high 186, batch 31 used:43
Sep 15 04:36:18 smtp2 snmpd[2489]: Connection from UDP:
[206.16.46.12]:59774
Sep 15 04:36:18 smtp2 kernel: cpu 1 cold: high 62, batch 15 used:13
Sep 15 04:36:18 smtp2 snmpd[2489]: Connection from UDP:
[206.16.46.12]:59774
Sep 15 04:36:18 smtp2 kernel: Node 0 Normal per-cpu: empty
Sep 15 04:36:18 smtp2 snmpd[2489]: Connection from UDP:
[206.16.46.12]:59774
Sep 15 04:36:18 smtp2 kernel: Node 0 HighMem per-cpu: empty
Sep 15 04:36:19 smtp2 snmpd[2489]: Connection from UDP:
[206.16.46.12]:59774
Sep 15 04:36:20 smtp2 kernel: Free pages:        8132kB (0kB HighMem)
Sep 15 04:36:20 smtp2 snmpd[2489]: Connection from UDP:
[206.16.46.12]:59774
Sep 15 04:36:22 smtp2 kernel: Active:195980 inactive:159056 dirty:0
writeback:0 unstable:0 free:2033 slab:32659 mapped-file:2736
mapped-anon:353032 pagetables:97169
Sep 15 04:36:24 smtp2 kernel: Node 0 DMA free:2152kB min:28kB low:32kB
high:40kB active:0kB inactive:0kB present:10716kB pages_scanned:0
all_unreclaimable? yes
Sep 15 04:36:39 smtp2 kernel: lowmem_reserve[]: 0 2003 2003 2003
Sep 15 04:37:20 smtp2 kernel: Node 0 DMA32 free:5980kB min:5708kB
low:7132kB high:8560kB active:783924kB inactive:636220kB
present:2051184kB pages_scanned:3003394 all_unreclaimable? yes
Sep 15 04:37:20 smtp2 kernel: lowmem_reserve[]: 0 0 0 0
Sep 15 04:37:20 smtp2 kernel: Node 0 Normal free:0kB min:0kB low:0kB
high:0kB active:0kB inactive:0kB present:0kB pages_scanned:0
all_unreclaimable? no
Sep 15 04:37:20 smtp2 kernel: lowmem_reserve[]: 0 0 0 0
Sep 15 04:37:20 smtp2 kernel: Node 0 HighMem free:0kB min:128kB
low:128kB high:128kB active:0kB inactive:0kB present:0kB pages_scanned:0
all_unreclaimable? no
Sep 15 04:37:20 smtp2 kernel: lowmem_reserve[]: 0 0 0 0
Sep 15 04:37:20 smtp2 kernel: Node 0 DMA: 2*4kB 4*8kB 2*16kB 5*32kB
4*64kB 1*128kB 2*256kB 0*512kB 1*1024kB 0*2048kB 0*4096kB = 2152kB
Sep 15 04:37:20 smtp2 kernel: Node 0 DMA32: 87*4kB 34*8kB 3*16kB 12*32kB
5*64kB 2*128kB 1*256kB 0*512kB 0*1024kB 0*2048kB 1*4096kB = 5980kB
Sep 15 04:37:20 smtp2 kernel: Node 0 Normal: empty
Sep 15 04:37:20 smtp2 kernel: Node 0 HighMem: empty
Sep 15 04:37:20 smtp2 kernel: 6591 pagecache pages
Sep 15 04:37:20 smtp2 kernel: Swap cache: add 619164, delete 615423,
find 100105/111253, race 3+10
Sep 15 04:37:20 smtp2 kernel: Free swap  = 0kB
Sep 15 04:37:20 smtp2 kernel: Total swap = 2096472kB
Sep 15 04:37:20 smtp2 kernel: Free swap:            0kB
Sep 15 04:37:20 smtp2 kernel: 524000 pages of RAM
Sep 15 04:37:20 smtp2 kernel: 9347 reserved pages
Sep 15 04:37:20 smtp2 kernel: 1055448 pages shared
Sep 15 04:37:20 smtp2 kernel: 3741 pages swap cached
Sep 15 04:37:20 smtp2 kernel: Out of memory: Killed process 3431 (crond).
Sep 15 04:38:12 smtp2 snmpd[2489]: Connection from UDP:
[206.16.46.12]:59774
Sep 15 04:38:14 smtp2 last message repeated 44 times
Sep 15 04:38:14 smtp2 snmpd[2489]: Connection from UDP:
[206.16.46.12]:53589
Sep 15 04:38:14 smtp2 snmpd[2489]: Received SNMP packet(s) from UDP:
[206.16.46.12]:53589
Sep 15 04:38:14 smtp2 snmpd[2489]: Connection from UDP:
[206.16.46.12]:53589
Sep 15 04:38:14 smtp2 last message repeated 130 times
Sep 15 04:38:14 smtp2 snmpd[2489]: Connection from UDP:
[206.16.46.12]:44340
Sep 15 04:38:14 smtp2 snmpd[2489]: Received SNMP packet(s) from UDP:
[206.16.46.12]:44340
Sep 15 04:38:14 smtp2 snmpd[2489]: Connection from UDP:
[206.16.46.12]:44340
Sep 15 04:38:14 smtp2 last message repeated 113 times
Sep 15 04:40:43 smtp2 kernel: modclusterd invoked oom-killer:
gfp_mask=0x201d2, order=0, oomkilladj=0
Sep 15 04:40:43 smtp2 kernel:
Sep 15 04:40:43 smtp2 kernel: Call Trace:
Sep 15 04:40:43 smtp2 kernel:  [<ffffffff800c3bda>] out_of_memory+0x8e/0x2f3
Sep 15 04:40:43 smtp2 kernel:  [<ffffffff8000f2ea>]
__alloc_pages+0x245/0x2ce
Sep 15 04:40:43 smtp2 kernel:  [<ffffffff80012a77>]
__do_page_cache_readahead+0xa2/0x1e6
Sep 15 04:40:43 smtp2 kernel:  [<ffffffff80063a05>]
__wait_on_bit_lock+0x5b/0x66
Sep 15 04:40:43 smtp2 kernel:  [<ffffffff8003f626>] __lock_page+0x5e/0x64
Sep 15 04:40:43 smtp2 kernel:  [<ffffffff80013406>]
filemap_nopage+0x148/0x322
Sep 15 04:40:43 smtp2 kernel:  [<ffffffff8000886a>]
__handle_mm_fault+0x1f8/0xe5c
Sep 15 04:40:43 smtp2 kernel:  [<ffffffff80066b9f>]
do_page_fault+0x4cb/0x830
Sep 15 04:40:43 smtp2 kernel:  [<ffffffff800320b8>] sys_faccessat+0xe4/0x18d
Sep 15 04:40:43 smtp2 kernel:  [<ffffffff8005dde9>] error_exit+0x0/0x84
Sep 15 04:40:43 smtp2 kernel:
Sep 15 04:40:43 smtp2 kernel: Mem-info:
Sep 15 04:40:43 smtp2 kernel: Node 0 DMA per-cpu:
Sep 15 04:40:43 smtp2 kernel: cpu 0 hot: high 0, batch 1 used:0
Sep 15 04:40:43 smtp2 kernel: cpu 0 cold: high 0, batch 1 used:0
Sep 15 04:40:43 smtp2 kernel: cpu 1 hot: high 0, batch 1 used:0
Sep 15 04:40:43 smtp2 kernel: cpu 1 cold: high 0, batch 1 used:0
Sep 15 04:40:43 smtp2 kernel: Node 0 DMA32 per-cpu:
Sep 15 04:40:43 smtp2 kernel: cpu 0 hot: high 186, batch 31 used:35
Sep 15 04:40:43 smtp2 kernel: cpu 0 cold: high 62, batch 15 used:4
Sep 15 04:40:43 smtp2 kernel: cpu 1 hot: high 186, batch 31 used:43
Sep 15 04:40:43 smtp2 kernel: cpu 1 cold: high 62, batch 15 used:14
Sep 15 04:40:43 smtp2 kernel: Node 0 Normal per-cpu: empty
Sep 15 04:40:43 smtp2 kernel: Node 0 HighMem per-cpu: empty
Sep 15 04:40:43 smtp2 kernel: Free pages:        7804kB (0kB HighMem)
Sep 15 04:40:43 smtp2 kernel: Active:144060 inactive:212032 dirty:7
writeback:0 unstable:0 free:1951 slab:32604 mapped-file:2742
mapped-anon:353322 pagetables:97083
Sep 15 04:40:43 smtp2 kernel: Node 0 DMA free:2152kB min:28kB low:32kB
high:40kB active:0kB inactive:0kB present:10716kB pages_scanned:0
all_unreclaimable? yes
Sep 15 04:40:43 smtp2 kernel: lowmem_reserve[]: 0 2003 2003 2003
Sep 15 04:40:43 smtp2 kernel: Node 0 DMA32 free:5652kB min:5708kB
low:7132kB high:8560kB active:576240kB inactive:848128kB
present:2051184kB pages_scanned:2766187 all_unreclaimable? yes
Sep 15 04:40:43 smtp2 kernel: lowmem_reserve[]: 0 0 0 0
Sep 15 04:40:43 smtp2 kernel: Node 0 Normal free:0kB min:0kB low:0kB
high:0kB active:0kB inactive:0kB present:0kB pages_scanned:0
all_unreclaimable? no
Sep 15 04:40:43 smtp2 kernel: lowmem_reserve[]: 0 0 0 0
Sep 15 04:40:43 smtp2 kernel: Node 0 HighMem free:0kB min:128kB
low:128kB high:128kB active:0kB inactive:0kB present:0kB pages_scanned:0
all_unreclaimable? no
Sep 15 04:40:43 smtp2 kernel: lowmem_reserve[]: 0 0 0 0
Sep 15 04:40:43 smtp2 kernel: Node 0 DMA: 2*4kB 4*8kB 2*16kB 5*32kB
4*64kB 1*128kB 2*256kB 0*512kB 1*1024kB 0*2048kB 0*4096kB = 2152kB
Sep 15 04:40:43 smtp2 kernel: Node 0 DMA32: 17*4kB 24*8kB 5*16kB 12*32kB
5*64kB 2*128kB 1*256kB 0*512kB 0*1024kB 0*2048kB 1*4096kB = 5652kB
Sep 15 04:40:43 smtp2 kernel: Node 0 Normal: empty
Sep 15 04:40:43 smtp2 kernel: Node 0 HighMem: empty
Sep 15 04:40:43 smtp2 kernel: 6698 pagecache pages
Sep 15 04:40:43 smtp2 kernel: Swap cache: add 626883, delete 623182,
find 100385/112417, race 3+15
Sep 15 04:40:43 smtp2 kernel: Free swap  = 0kB
Sep 15 04:40:43 smtp2 kernel: Total swap = 2096472kB
Sep 15 04:40:43 smtp2 kernel: Free swap:            0kB
Sep 15 04:40:43 smtp2 kernel: 524000 pages of RAM
Sep 15 04:40:43 smtp2 kernel: 9347 reserved pages
Sep 15 04:40:43 smtp2 kernel: 1056040 pages shared
Sep 15 04:40:43 smtp2 kernel: 3701 pages swap cached
Sep 15 04:40:43 smtp2 kernel: Out of memory: Killed process 3449 (crond).
Sep 15 04:40:52 smtp2 snmpd[2489]: Connection from UDP:
[206.16.46.12]:50576
Sep 15 04:40:52 smtp2 snmpd[2489]: Received SNMP packet(s) from UDP:
[206.16.46.12]:50576
Sep 15 04:40:53 smtp2 snmpd[2489]: Connection from UDP:
[206.16.46.12]:50576
Sep 15 04:40:53 smtp2 last message repeated 60 times
Sep 15 04:43:46 smtp2 kernel: postdrop invoked oom-killer:
gfp_mask=0x201d2, order=0, oomkilladj=0
Sep 15 04:43:56 smtp2 kernel:
Sep 15 04:43:56 smtp2 kernel: Call Trace:
Sep 15 04:43:56 smtp2 kernel:  [<ffffffff800c3bda>] out_of_memory+0x8e/0x2f3
Sep 15 04:43:56 smtp2 kernel:  [<ffffffff8000f2ea>]
__alloc_pages+0x245/0x2ce
Sep 15 04:43:56 smtp2 kernel:  [<ffffffff80012a77>]
__do_page_cache_readahead+0xa2/0x1e6
Sep 15 04:43:56 smtp2 kernel:  [<ffffffff80063a05>]
__wait_on_bit_lock+0x5b/0x66
Sep 15 04:43:56 smtp2 kernel:  [<ffffffff8003f626>] __lock_page+0x5e/0x64
Sep 15 04:43:56 smtp2 kernel:  [<ffffffff80013406>]
filemap_nopage+0x148/0x322
Sep 15 04:43:56 smtp2 kernel:  [<ffffffff8000886a>]
__handle_mm_fault+0x1f8/0xe5c
Sep 15 04:43:56 smtp2 kernel:  [<ffffffff80066b9f>]
do_page_fault+0x4cb/0x830
Sep 15 04:43:56 smtp2 kernel:  [<ffffffff80059fdf>] hrtimer_cancel+0xc/0x16
Sep 15 04:43:56 smtp2 kernel:  [<ffffffff80063db7>] do_nanosleep+0x47/0x70
Sep 15 04:43:57 smtp2 kernel:  [<ffffffff80059ecc>]
hrtimer_nanosleep+0x58/0x118
Sep 15 04:43:57 smtp2 kernel:  [<ffffffff8005dde9>] error_exit+0x0/0x84
Sep 15 04:43:57 smtp2 kernel:
Sep 15 04:43:57 smtp2 kernel: Mem-info:
Sep 15 04:43:57 smtp2 kernel: Node 0 DMA per-cpu:
Sep 15 04:43:57 smtp2 kernel: cpu 0 hot: high 0, batch 1 used:0
Sep 15 04:43:57 smtp2 kernel: cpu 0 cold: high 0, batch 1 used:0
Sep 15 04:43:57 smtp2 kernel: cpu 1 hot: high 0, batch 1 used:0
Sep 15 04:43:57 smtp2 kernel: cpu 1 cold: high 0, batch 1 used:0
Sep 15 04:43:57 smtp2 kernel: Node 0 DMA32 per-cpu:
Sep 15 04:43:57 smtp2 kernel: cpu 0 hot: high 186, batch 31 used:41
Sep 15 04:43:57 smtp2 kernel: cpu 0 cold: high 62, batch 15 used:60
Sep 15 04:43:57 smtp2 kernel: cpu 1 hot: high 186, batch 31 used:14
Sep 15 04:43:57 smtp2 kernel: cpu 1 cold: high 62, batch 15 used:60
Sep 15 04:43:57 smtp2 kernel: Node 0 Normal per-cpu: empty
Sep 15 04:43:58 smtp2 kernel: Node 0 HighMem per-cpu: empty
Sep 15 04:43:58 smtp2 kernel: Free pages:        7816kB (0kB HighMem)
Sep 15 04:43:58 smtp2 kernel: Active:224553 inactive:131064 dirty:1
writeback:0 unstable:0 free:1954 slab:32705 mapped-file:2719
mapped-anon:353434 pagetables:96958
Sep 15 04:43:58 smtp2 kernel: Node 0 DMA free:2152kB min:28kB low:32kB
high:40kB active:0kB inactive:0kB present:10716kB pages_scanned:0
all_unreclaimable? yes
Sep 15 04:43:58 smtp2 kernel: lowmem_reserve[]: 0 2003 2003 2003
Sep 15 04:43:58 smtp2 kernel: Node 0 DMA32 free:5664kB min:5708kB
low:7132kB high:8560kB active:898120kB inactive:524348kB
present:2051184kB pages_scanned:2946614 all_unreclaimable? yes
Sep 15 04:43:58 smtp2 kernel: lowmem_reserve[]: 0 0 0 0
Sep 15 04:43:58 smtp2 kernel: Node 0 Normal free:0kB min:0kB low:0kB
high:0kB active:0kB inactive:0kB present:0kB pages_scanned:0
all_unreclaimable? no
Sep 15 04:43:58 smtp2 kernel: lowmem_reserve[]: 0 0 0 0
Sep 15 04:43:58 smtp2 kernel: Node 0 HighMem free:0kB min:128kB
low:128kB high:128kB active:0kB inactive:0kB present:0kB pages_scanned:0
all_unreclaimable? no
Sep 15 04:43:58 smtp2 kernel: lowmem_reserve[]: 0 0 0 0
Sep 15 04:43:58 smtp2 kernel: Node 0 DMA: 2*4kB 4*8kB 2*16kB 5*32kB
4*64kB 1*128kB 2*256kB 0*512kB 1*1024kB 0*2048kB 0*4096kB = 2152kB
Sep 15 04:43:58 smtp2 kernel: Node 0 DMA32: 14*4kB 27*8kB 5*16kB 12*32kB
5*64kB 2*128kB 1*256kB 0*512kB 0*1024kB 0*2048kB 1*4096kB = 5664kB
Sep 15 04:43:58 smtp2 kernel: Node 0 Normal: empty
Sep 15 04:43:58 smtp2 kernel: Node 0 HighMem: empty
Sep 15 04:43:58 smtp2 kernel: 6522 pagecache pages
Sep 15 04:43:58 smtp2 kernel: Swap cache: add 630480, delete 626780,
find 100415/112836, race 3+15
Sep 15 04:43:58 smtp2 kernel: Free swap  = 0kB
Sep 15 04:43:58 smtp2 kernel: Total swap = 2096472kB
Sep 15 04:43:58 smtp2 kernel: Free swap:            0kB
Sep 15 04:43:58 smtp2 kernel: 524000 pages of RAM
Sep 15 04:43:58 smtp2 kernel: 9347 reserved pages
Sep 15 04:43:58 smtp2 kernel: 1052953 pages shared
Sep 15 04:43:58 smtp2 kernel: 3700 pages swap cached
Sep 15 04:43:58 smtp2 kernel: Out of memory: Killed process 3467 (crond).
Sep 15 04:44:38 smtp2 kernel: postdrop invoked oom-killer:
gfp_mask=0x201d2, order=0, oomkilladj=0
Sep 15 04:45:24 smtp2 snmpd[2489]: Connection from UDP:
[206.16.46.12]:43992
Sep 15 04:45:24 smtp2 kernel:
Sep 15 04:45:25 smtp2 kernel: Call Trace:
Sep 15 04:45:25 smtp2 kernel:  [<ffffffff800c3bda>] out_of_memory+0x8e/0x2f3
Sep 15 04:45:25 smtp2 kernel:  [<ffffffff8000f2ea>]
__alloc_pages+0x245/0x2ce
Sep 15 04:45:25 smtp2 kernel:  [<ffffffff80012a77>]
__do_page_cache_readahead+0xa2/0x1e6
Sep 15 04:45:25 smtp2 kernel:  [<ffffffff80054ab8>] sock_sendmsg+0xf3/0x110
Sep 15 04:45:25 smtp2 kernel:  [<ffffffff8000e87a>] link_path_walk+0xd3/0xe5
Sep 15 04:45:25 smtp2 kernel:  [<ffffffff80013406>]
filemap_nopage+0x148/0x322
Sep 15 04:45:25 smtp2 kernel:  [<ffffffff8000886a>]
__handle_mm_fault+0x1f8/0xe5c
Sep 15 04:45:25 smtp2 kernel:  [<ffffffff80066b9f>]
do_page_fault+0x4cb/0x830
Sep 15 04:45:25 smtp2 kernel:  [<ffffffff80059fdf>] hrtimer_cancel+0xc/0x16
Sep 15 04:45:25 smtp2 kernel:  [<ffffffff80063db7>] do_nanosleep+0x47/0x70
Sep 15 04:45:25 smtp2 kernel:  [<ffffffff80059ecc>]
hrtimer_nanosleep+0x58/0x118
Sep 15 04:45:25 smtp2 kernel:  [<ffffffff8005dde9>] error_exit+0x0/0x84
Sep 15 04:45:25 smtp2 kernel:
Sep 15 04:45:25 smtp2 kernel: Mem-info:
Sep 15 04:45:25 smtp2 kernel: Node 0 DMA per-cpu:
Sep 15 04:45:25 smtp2 kernel: cpu 0 hot: high 0, batch 1 used:0
Sep 15 04:45:25 smtp2 kernel: cpu 0 cold: high 0, batch 1 used:0
Sep 15 04:45:25 smtp2 kernel: cpu 1 hot: high 0, batch 1 used:0
Sep 15 04:45:25 smtp2 kernel: cpu 1 cold: high 0, batch 1 used:0
Sep 15 04:45:25 smtp2 kernel: Node 0 DMA32 per-cpu:
Sep 15 04:45:25 smtp2 kernel: cpu 0 hot: high 186, batch 31 used:16
Sep 15 04:45:25 smtp2 kernel: cpu 0 cold: high 62, batch 15 used:14
Sep 15 04:45:25 smtp2 kernel: cpu 1 hot: high 186, batch 31 used:12
Sep 15 04:45:25 smtp2 kernel: cpu 1 cold: high 62, batch 15 used:40
Sep 15 04:45:25 smtp2 kernel: Node 0 Normal per-cpu: empty
Sep 15 04:45:25 smtp2 kernel: Node 0 HighMem per-cpu: empty
Sep 15 04:45:25 smtp2 kernel: Free pages:        8072kB (0kB HighMem)
Sep 15 04:45:25 smtp2 kernel: Active:196819 inactive:156715 dirty:0
writeback:0 unstable:0 free:2018 slab:32583 mapped-file:2736
mapped-anon:353474 pagetables:96986
Sep 15 04:45:25 smtp2 kernel: Node 0 DMA free:2152kB min:28kB low:32kB
high:40kB active:0kB inactive:0kB present:10716kB pages_scanned:0
all_unreclaimable? yes
Sep 15 04:45:25 smtp2 kernel: lowmem_reserve[]: 0 2003 2003 2003
Sep 15 04:45:25 smtp2 kernel: Node 0 DMA32 free:5920kB min:5708kB
low:7132kB high:8560kB active:787276kB inactive:626860kB
present:2051184kB pages_scanned:3225864 all_unreclaimable? yes
Sep 15 04:45:25 smtp2 kernel: lowmem_reserve[]: 0 0 0 0
Sep 15 04:45:25 smtp2 kernel: Node 0 Normal free:0kB min:0kB low:0kB
high:0kB active:0kB inactive:0kB present:0kB pages_scanned:0
all_unreclaimable? no
Sep 15 04:45:25 smtp2 kernel: lowmem_reserve[]: 0 0 0 0
Sep 15 04:45:25 smtp2 kernel: Node 0 HighMem free:0kB min:128kB
low:128kB high:128kB active:0kB inactive:0kB present:0kB pages_scanned:0
all_unreclaimable? no
Sep 15 04:45:25 smtp2 kernel: lowmem_reserve[]: 0 0 0 0
Sep 15 04:45:25 smtp2 kernel: Node 0 DMA: 2*4kB 4*8kB 2*16kB 5*32kB
4*64kB 1*128kB 2*256kB 0*512kB 1*1024kB 0*2048kB 0*4096kB = 2152kB
Sep 15 04:45:25 smtp2 kernel: Node 0 DMA32: 80*4kB 26*8kB 5*16kB 12*32kB
5*64kB 2*128kB 1*256kB 0*512kB 0*1024kB 0*2048kB 1*4096kB = 5920kB
Sep 15 04:45:25 smtp2 kernel: Node 0 Normal: empty
Sep 15 04:45:25 smtp2 kernel: Node 0 HighMem: empty
Sep 15 04:45:25 smtp2 kernel: 6528 pagecache pages
Sep 15 04:45:25 smtp2 kernel: Swap cache: add 630994, delete 627294,
find 100446/112908, race 3+15
Sep 15 04:45:25 smtp2 kernel: Free swap  = 0kB
Sep 15 04:45:25 smtp2 kernel: Total swap = 2096472kB
Sep 15 04:45:25 smtp2 kernel: Free swap:            0kB
Sep 15 04:45:25 smtp2 kernel: 524000 pages of RAM
Sep 15 04:45:25 smtp2 kernel: 9347 reserved pages
Sep 15 04:45:25 smtp2 kernel: 1055739 pages shared
Sep 15 04:45:25 smtp2 kernel: 3700 pages swap cached




Andrea Dell'Amico wrote:
> On Wed, 2009-09-16 at 03:23 -0700, David Buechler wrote:
>   
>> Hi Andrea!
>>
>> Heh, yeah, I should have specified that.  No, I'm not using Xen on this 
>> particular system.  I'm using Xen on other systems under DRBD 8.2 and 
>> they're pretty stable.
>>     
>
> Ugh. A new kind of crash, not reassuring :-).
> I think you should post the kernel Oops on the list, then.
>
>   
>> -- 
>> Regards,
>> David A. Buechler
>> System Administrator,
>> CableAmerica Missouri
>>     
>
> ciao
> andrea
>
>   




-- 
This message has been scanned for viruses and
dangerous content by MailScanner, and is
believed to be clean.




More information about the drbd-user mailing list