Note: "permalinks" may not be as permanent as we would like,
direct links of old sources may well be a few messages off.
Hi,
no one has a clue on this?
I've tried rising some kernel net params - without success.
node1:~# sysctl -a| egrep '(net.core.rmem_max|net.core.wmem_max|net.ipv4.tcp_rmem|net.ipv4.tcp_wmem|net.ipv4.tcp_mem)'
net.core.wmem_max = 10485760
net.core.rmem_max = 10485760
net.ipv4.tcp_mem = 128000 170666 256000
net.ipv4.tcp_wmem = 65536 65536 10485760
net.ipv4.tcp_rmem = 65536 87380 10485760
--
best regards,
Konrad Korzeniowski
[2011-03-13 16:38] Konrad Korzeniowski:
> Hi,
>
> I hope that some will be able to help me with strange (at least for me) issue after Debian upgrade (Lenny -> Squeeze).
>
> I'm using dual master/slave setup controlled by heartbeat, where the ha resources are whole vservers.
> There is dedicated drbd/heartbeat crossover connection between nodes.
>
> My current drbd.conf:
> -------------------------------------------------------------------------------------------
> global {
> usage-count no;
> }
> common {
> syncer { rate 10M; }
> }
> resource ocfs2 {
> protocol C;
> handlers {
> pri-on-incon-degr "echo o > /proc/sysrq-trigger ; halt -f";
> pri-lost-after-sb "echo o > /proc/sysrq-trigger ; halt -f";
> local-io-error "echo o > /proc/sysrq-trigger ; halt -f";
> }
> startup {
> }
> disk {
> on-io-error detach;
> }
> net {
> allow-two-primaries;
> after-sb-0pri disconnect;
> after-sb-1pri disconnect;
> after-sb-2pri disconnect;
> rr-conflict disconnect;
> }
> syncer {
> rate 10M;
> al-extents 257;
> }
> on node1 {
> device /dev/drbd0;
> disk /dev/sda6;
> address 10.10.10.45:7788;
> meta-disk internal;
> }
> on node2 {
> device /dev/drbd0;
> disk /dev/sda6;
> address 10.10.10.46:7788;
> meta-disk internal;
> }
> }
> resource grupa1-200 {
> protocol C;
> handlers {
> pri-on-incon-degr "echo o > /proc/sysrq-trigger ; halt -f";
> pri-lost-after-sb "echo o > /proc/sysrq-trigger ; halt -f";
> local-io-error "echo o > /proc/sysrq-trigger ; halt -f";
> }
> startup {
> }
> disk {
> on-io-error detach;
> }
> net {
> after-sb-0pri disconnect;
> after-sb-1pri disconnect;
> after-sb-2pri disconnect;
> rr-conflict disconnect;
> # wartosc 0 oznacza auto
> sndbuf-size 0;
> }
> syncer {
> rate 30M;
> al-extents 257;
> }
> on node1 {
> device /dev/drbd1;
> disk /dev/sda7;
> address 10.10.10.45:7789;
> meta-disk internal;
> }
> on node2 {
> device /dev/drbd1;
> disk /dev/sda7;
> address 10.10.10.46:7789;
> meta-disk internal;
> }
> }
> resource grupa2-200 {
> protocol C;
> handlers {
> pri-on-incon-degr "echo o > /proc/sysrq-trigger ; halt -f";
> pri-lost-after-sb "echo o > /proc/sysrq-trigger ; halt -f";
> local-io-error "echo o > /proc/sysrq-trigger ; halt -f";
> }
> startup {
> }
> disk {
> on-io-error detach;
> }
> net {
> after-sb-0pri disconnect;
> after-sb-1pri disconnect;
> after-sb-2pri disconnect;
> rr-conflict disconnect;
> }
> syncer {
> rate 30M;
> al-extents 257;
> }
> on node1 {
> device /dev/drbd2;
> disk /dev/sdb6;
> address 10.10.10.45:7790;
> meta-disk internal;
> }
> on node2 {
> device /dev/drbd2;
> disk /dev/sdb6;
> address 10.10.10.46:7790;
> meta-disk internal;
> }
> }
> -------------------------------------------------------------------------------------------
>
> And the problem is that every morning (1 or more times) in logs I can see:
>
> Mar 13 06:30:35 node1 kernel: [902670.893173] __ratelimit: 407 callbacks suppressed
> Mar 13 06:30:35 node1 kernel: [902670.893175] drbd1_receiver: page allocation failure. order:4, mode:0x4020
> Mar 13 06:30:35 node1 kernel: [902670.893178] Pid: 25245, comm: drbd1_receiver Not tainted 2.6.32-5-vserver-686-bigmem #1
> Mar 13 06:30:35 node1 kernel: [902670.893180] Call Trace:
> Mar 13 06:30:35 node1 kernel: [902670.893187] [<c109ad8c>] ? __alloc_pages_nodemask+0x484/0x4d9
> Mar 13 06:30:35 node1 kernel: [902670.893190] [<c109aded>] ? __get_free_pages+0xc/0x17
> Mar 13 06:30:35 node1 kernel: [902670.893194] [<c10bf261>] ? __kmalloc+0x30/0x128
> Mar 13 06:30:35 node1 kernel: [902670.893207] [<c11ee2f4>] ? pskb_expand_head+0x4f/0x157
> Mar 13 06:30:35 node1 kernel: [902670.893209] [<c11ee74b>] ? __pskb_pull_tail+0x40/0x200
> Mar 13 06:30:35 node1 kernel: [902670.893212] [<c11f555e>] ? dev_queue_xmit+0xe4/0x38e
> Mar 13 06:30:35 node1 kernel: [902670.893215] [<c1216cea>] ? ip_finish_output2+0x187/0x1c2
> Mar 13 06:30:35 node1 kernel: [902670.893217] [<c12161ed>] ? ip_local_out+0x15/0x17
> Mar 13 06:30:35 node1 kernel: [902670.893219] [<c12169cc>] ? ip_queue_xmit+0x31d/0x378
> Mar 13 06:30:35 node1 kernel: [902670.893222] [<c1224fdf>] ? tcp_established_options+0x1d/0x8b
> Mar 13 06:30:35 node1 kernel: [902670.893224] [<c1225c52>] ? tcp_transmit_skb+0x595/0x5cc
> Mar 13 06:30:35 node1 kernel: [902670.893227] [<c1227cb5>] ? tcp_write_xmit+0x7a3/0x874
> Mar 13 06:30:35 node1 kernel: [902670.893229] [<c12236e4>] ? tcp_ack+0x1611/0x1802
> Mar 13 06:30:35 node1 kernel: [902670.893231] [<c1224fdf>] ? tcp_established_options+0x1d/0x8b
> Mar 13 06:30:35 node1 kernel: [902670.893233] [<c12252a3>] ? tcp_current_mss+0x38/0x53
> Mar 13 06:30:35 node1 kernel: [902670.893236] [<c1227dcc>] ? __tcp_push_pending_frames+0x1e/0x50
> Mar 13 06:30:35 node1 kernel: [902670.893238] [<c12238f0>] ? tcp_data_snd_check+0x1b/0xd2
> Mar 13 06:30:35 node1 kernel: [902670.893240] [<c1224493>] ? tcp_rcv_established+0x5d6/0x62c
> Mar 13 06:30:35 node1 kernel: [902670.893243] [<c128afe9>] ? schedule_timeout+0x20/0xb0
> Mar 13 06:30:35 node1 kernel: [902670.893245] [<c122a9ae>] ? tcp_v4_do_rcv+0x15f/0x2cf
> Mar 13 06:30:35 node1 kernel: [902670.893248] [<c11eabce>] ? sk_wait_data+0x99/0xa2
> Mar 13 06:30:35 node1 kernel: [902670.893250] [<c121bf64>] ? tcp_prequeue_process+0x57/0x6b
> Mar 13 06:30:35 node1 kernel: [902670.893253] [<c121cf66>] ? tcp_recvmsg+0x441/0x8e5
> Mar 13 06:30:35 node1 kernel: [902670.893255] [<c11e9cdf>] ? sock_common_recvmsg+0x2f/0x45
> Mar 13 06:30:35 node1 kernel: [902670.893257] [<c11e9cdf>] ? sock_common_recvmsg+0x2f/0x45
> Mar 13 06:30:35 node1 kernel: [902670.893260] [<c11e7fc3>] ? __sock_recvmsg+0x50/0xbb
> Mar 13 06:30:35 node1 kernel: [902670.893262] [<c11e855d>] ? sock_recvmsg+0x9d/0xb6
> Mar 13 06:30:35 node1 kernel: [902670.893266] [<c104b08a>] ? autoremove_wake_function+0x0/0x2d
> Mar 13 06:30:35 node1 kernel: [902670.893269] [<c113c658>] ? __make_request+0x364/0x3d9
> Mar 13 06:30:35 node1 kernel: [902670.893276] [<f9067ddf>] ? drbd_recv+0x62/0x106 [drbd]
> Mar 13 06:30:35 node1 kernel: [902670.893280] [<f9067e95>] ? drbd_recv_header+0x12/0xa5 [drbd]
> Mar 13 06:30:35 node1 kernel: [902670.893285] [<f906e669>] ? drbdd_init+0xf3/0x245 [drbd]
> Mar 13 06:30:35 node1 kernel: [902670.893290] [<f9079bf7>] ? drbd_thread_setup+0x1f/0xcc [drbd]
> Mar 13 06:30:35 node1 kernel: [902670.893294] [<f9079bd8>] ? drbd_thread_setup+0x0/0xcc [drbd]
> Mar 13 06:30:35 node1 kernel: [902670.893296] [<c104ae58>] ? kthread+0x61/0x66
> Mar 13 06:30:35 node1 kernel: [902670.893299] [<c104adf7>] ? kthread+0x0/0x66
> Mar 13 06:30:35 node1 kernel: [902670.893302] [<c1008d87>] ? kernel_thread_helper+0x7/0x10
> Mar 13 06:30:35 node1 kernel: [902670.893303] Mem-Info:
> Mar 13 06:30:35 node1 kernel: [902670.893304] DMA per-cpu:
> Mar 13 06:30:35 node1 kernel: [902670.893306] CPU 0: hi: 0, btch: 1 usd: 0
> Mar 13 06:30:35 node1 kernel: [902670.893307] CPU 1: hi: 0, btch: 1 usd: 0
> Mar 13 06:30:35 node1 kernel: [902670.893309] Normal per-cpu:
> Mar 13 06:30:35 node1 kernel: [902670.893310] CPU 0: hi: 186, btch: 31 usd: 173
> Mar 13 06:30:35 node1 kernel: [902670.893312] CPU 1: hi: 186, btch: 31 usd: 155
> Mar 13 06:30:35 node1 kernel: [902670.893313] HighMem per-cpu:
> Mar 13 06:30:35 node1 kernel: [902670.893314] CPU 0: hi: 186, btch: 31 usd: 47
> Mar 13 06:30:35 node1 kernel: [902670.893316] CPU 1: hi: 186, btch: 31 usd: 28
> Mar 13 06:30:35 node1 kernel: [902670.893319] active_anon:137306 inactive_anon:39520 isolated_anon:0
> Mar 13 06:30:35 node1 kernel: [902670.893320] active_file:223625 inactive_file:375413 isolated_file:32
> Mar 13 06:30:35 node1 kernel: [902670.893321] unevictable:4083 dirty:1708 writeback:0 unstable:0
> Mar 13 06:30:35 node1 kernel: [902670.893321] free:187603 slab_reclaimable:50120 slab_unreclaimable:3975
> Mar 13 06:30:35 node1 kernel: [902670.893322] mapped:23637 shmem:11538 pagetables:2294 bounce:27
> Mar 13 06:30:35 node1 kernel: [902670.893327] DMA free:3524kB min:64kB low:80kB high:96kB active_anon:0kB inactive_anon:0kB active_file:2728kB
> inactive_file:6180kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15868kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB
> slab_reclaimable:3512kB slab_unreclaimable:8kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0
> all_unreclaimable? no
> Mar 13 06:30:35 node1 kernel: [902670.893331] lowmem_reserve[]: 0 865 4030 4030
> Mar 13 06:30:35 node1 kernel: [902670.893337] Normal free:40292kB min:3728kB low:4660kB high:5592kB active_anon:4280kB inactive_anon:15772kB
> active_file:199036kB inactive_file:355444kB unevictable:0kB isolated(anon):0kB isolated(file):128kB present:885944kB mlocked:0kB dirty:228kB
> writeback:0kB mapped:24kB shmem:8kB slab_reclaimable:196968kB slab_unreclaimable:15892kB kernel_stack:2608kB pagetables:104kB unstable:0kB
> bounce:108kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
> Mar 13 06:30:35 node1 kernel: [902670.893341] lowmem_reserve[]: 0 0 25321 25321
> Mar 13 06:30:35 node1 kernel: [902670.893347] HighMem free:706596kB min:512kB low:3920kB high:7332kB active_anon:544944kB inactive_anon:142308kB
> active_file:692736kB inactive_file:1140028kB unevictable:16332kB isolated(anon):0kB isolated(file):0kB present:3241132kB mlocked:16332kB dirty:6604kB
> writeback:0kB mapped:94524kB shmem:46144kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:9072kB unstable:0kB bounce:0kB
> writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
> Mar 13 06:30:35 node1 kernel: [902670.893351] lowmem_reserve[]: 0 0 0 0
> Mar 13 06:30:35 node1 kernel: [902670.893354] DMA: 15*4kB 15*8kB 9*16kB 22*32kB 7*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 1*2048kB 0*4096kB = 3524kB
> Mar 13 06:30:35 node1 kernel: [902670.893360] Normal: 3507*4kB 3161*8kB 37*16kB 8*32kB 2*64kB 1*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 40420kB
> Mar 13 06:30:35 node1 kernel: [902670.893367] HighMem: 77*4kB 56*8kB 23*16kB 12*32kB 2061*64kB 2550*128kB 704*256kB 104*512kB 13*1024kB 0*2048kB
> 0*4096kB = 706596kB
> Mar 13 06:30:35 node1 kernel: [902670.893373] 612114 total pagecache pages
> Mar 13 06:30:35 node1 kernel: [902670.893374] 160 pages in swap cache
> Mar 13 06:30:35 node1 kernel: [902670.893376] Swap cache stats: add 21929, delete 21769, find 243894/245286
> Mar 13 06:30:35 node1 kernel: [902670.893377] Free swap = 4880236kB
> Mar 13 06:30:35 node1 kernel: [902670.893378] Total swap = 4883640kB
> Mar 13 06:30:35 node1 kernel: [902670.902900] 1245184 pages RAM
> Mar 13 06:30:35 node1 kernel: [902670.902902] 1017858 pages HighMem
> Mar 13 06:30:35 node1 kernel: [902670.902903] 210949 pages reserved
> Mar 13 06:30:35 node1 kernel: [902670.902904] 327380 pages shared
> Mar 13 06:30:35 node1 kernel: [902670.902905] 642241 pages non-shared
> Mar 13 06:30:35 node1 kernel: [902670.902921] drbd1_receiver: page allocation failure. order:4, mode:0x4020
> Mar 13 06:30:35 node1 kernel: [902670.902924] Pid: 25245, comm: drbd1_receiver Not tainted 2.6.32-5-vserver-686-bigmem #1
> Mar 13 06:30:35 node1 kernel: [902670.902926] Call Trace:
> Mar 13 06:30:35 node1 kernel: [902670.902931] [<c109ad8c>] ? __alloc_pages_nodemask+0x484/0x4d9
> Mar 13 06:30:35 node1 kernel: [902670.902934] [<c109aded>] ? __get_free_pages+0xc/0x17
> Mar 13 06:30:35 node1 kernel: [902670.902937] [<c10bf261>] ? __kmalloc+0x30/0x128
> Mar 13 06:30:35 node1 kernel: [902670.902946] [<c11ee2f4>] ? pskb_expand_head+0x4f/0x157
> Mar 13 06:30:35 node1 kernel: [902670.902948] [<c11ee74b>] ? __pskb_pull_tail+0x40/0x200
> Mar 13 06:30:35 node1 kernel: [902670.902950] [<c11eb077>] ? sock_wfree+0x17/0x4b
> Mar 13 06:30:35 node1 kernel: [902670.902953] [<c11f555e>] ? dev_queue_xmit+0xe4/0x38e
> Mar 13 06:30:35 node1 kernel: [902670.902956] [<c1216cea>] ? ip_finish_output2+0x187/0x1c2
> Mar 13 06:30:35 node1 kernel: [902670.902958] [<c12161ed>] ? ip_local_out+0x15/0x17
> Mar 13 06:30:35 node1 kernel: [902670.902961] [<c12169cc>] ? ip_queue_xmit+0x31d/0x378
> Mar 13 06:30:35 node1 kernel: [902670.902963] [<c1224fdf>] ? tcp_established_options+0x1d/0x8b
> Mar 13 06:30:35 node1 kernel: [902670.902966] [<c1225c52>] ? tcp_transmit_skb+0x595/0x5cc
> Mar 13 06:30:35 node1 kernel: [902670.902968] [<c1227cb5>] ? tcp_write_xmit+0x7a3/0x874
> Mar 13 06:30:35 node1 kernel: [902670.902970] [<c12236e4>] ? tcp_ack+0x1611/0x1802
> Mar 13 06:30:35 node1 kernel: [902670.902973] [<c1224fdf>] ? tcp_established_options+0x1d/0x8b
> Mar 13 06:30:35 node1 kernel: [902670.902975] [<c12252a3>] ? tcp_current_mss+0x38/0x53
> Mar 13 06:30:35 node1 kernel: [902670.902977] [<c1227dcc>] ? __tcp_push_pending_frames+0x1e/0x50
> Mar 13 06:30:35 node1 kernel: [902670.902979] [<c12238f0>] ? tcp_data_snd_check+0x1b/0xd2
> Mar 13 06:30:35 node1 kernel: [902670.902982] [<c1223f8f>] ? tcp_rcv_established+0xd2/0x62c
> Mar 13 06:30:35 node1 kernel: [902670.902986] [<c128afe9>] ? schedule_timeout+0x20/0xb0
> Mar 13 06:30:35 node1 kernel: [902670.902989] [<c122a9ae>] ? tcp_v4_do_rcv+0x15f/0x2cf
> Mar 13 06:30:35 node1 kernel: [902670.902991] [<c11eabce>] ? sk_wait_data+0x99/0xa2
> Mar 13 06:30:35 node1 kernel: [902670.902993] [<c121bf64>] ? tcp_prequeue_process+0x57/0x6b
> Mar 13 06:30:35 node1 kernel: [902670.902996] [<c121cf66>] ? tcp_recvmsg+0x441/0x8e5
> Mar 13 06:30:35 node1 kernel: [902670.902998] [<c11e9cdf>] ? sock_common_recvmsg+0x2f/0x45
> Mar 13 06:30:35 node1 kernel: [902670.903000] [<c11e9cdf>] ? sock_common_recvmsg+0x2f/0x45
> Mar 13 06:30:35 node1 kernel: [902670.903003] [<c11e7fc3>] ? __sock_recvmsg+0x50/0xbb
> Mar 13 06:30:35 node1 kernel: [902670.903005] [<c11e855d>] ? sock_recvmsg+0x9d/0xb6
> Mar 13 06:30:35 node1 kernel: [902670.903009] [<c104b08a>] ? autoremove_wake_function+0x0/0x2d
> Mar 13 06:30:35 node1 kernel: [902670.903012] [<c113c658>] ? __make_request+0x364/0x3d9
> Mar 13 06:30:35 node1 kernel: [902670.903019] [<f9067ddf>] ? drbd_recv+0x62/0x106 [drbd]
> Mar 13 06:30:35 node1 kernel: [902670.903023] [<f9067e95>] ? drbd_recv_header+0x12/0xa5 [drbd]
> Mar 13 06:30:35 node1 kernel: [902670.903028] [<f906e669>] ? drbdd_init+0xf3/0x245 [drbd]
> Mar 13 06:30:35 node1 kernel: [902670.903033] [<f9079bf7>] ? drbd_thread_setup+0x1f/0xcc [drbd]
> Mar 13 06:30:35 node1 kernel: [902670.903037] [<f9079bd8>] ? drbd_thread_setup+0x0/0xcc [drbd]
> Mar 13 06:30:35 node1 kernel: [902670.903039] [<c104ae58>] ? kthread+0x61/0x66
> Mar 13 06:30:35 node1 kernel: [902670.903041] [<c104adf7>] ? kthread+0x0/0x66
> Mar 13 06:30:35 node1 kernel: [902670.903044] [<c1008d87>] ? kernel_thread_helper+0x7/0x10
> Mar 13 06:30:35 node1 kernel: [902670.903046] Mem-Info:
> Mar 13 06:30:35 node1 kernel: [902670.903048] DMA per-cpu:
> Mar 13 06:30:35 node1 kernel: [902670.903049] CPU 0: hi: 0, btch: 1 usd: 0
> Mar 13 06:30:35 node1 kernel: [902670.903051] CPU 1: hi: 0, btch: 1 usd: 0
> Mar 13 06:30:35 node1 kernel: [902670.903052] Normal per-cpu:
> Mar 13 06:30:35 node1 kernel: [902670.903054] CPU 0: hi: 186, btch: 31 usd: 173
> Mar 13 06:30:35 node1 kernel: [902670.903055] CPU 1: hi: 186, btch: 31 usd: 154
> Mar 13 06:30:35 node1 kernel: [902670.903057] HighMem per-cpu:
> Mar 13 06:30:35 node1 kernel: [902670.903058] CPU 0: hi: 186, btch: 31 usd: 47
> Mar 13 06:30:35 node1 kernel: [902670.903060] CPU 1: hi: 186, btch: 31 usd: 20
> Mar 13 06:30:35 node1 kernel: [902670.903063] active_anon:137306 inactive_anon:39520 isolated_anon:0
> Mar 13 06:30:35 node1 kernel: [902670.903064] active_file:223625 inactive_file:375413 isolated_file:32
> Mar 13 06:30:35 node1 kernel: [902670.903065] unevictable:4083 dirty:1708 writeback:0 unstable:0
> Mar 13 06:30:35 node1 kernel: [902670.903066] free:187603 slab_reclaimable:50120 slab_unreclaimable:3975
> Mar 13 06:30:35 node1 kernel: [902670.903066] mapped:23637 shmem:11538 pagetables:2294 bounce:27
> Mar 13 06:30:35 node1 kernel: [902670.903071] DMA free:3524kB min:64kB low:80kB high:96kB active_anon:0kB inactive_anon:0kB active_file:2728kB
> inactive_file:6180kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15868kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB
> slab_reclaimable:3512kB slab_unreclaimable:8kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0
> all_unreclaimable? no
> Mar 13 06:30:35 node1 kernel: [902670.903075] lowmem_reserve[]: 0 865 4030 4030
> Mar 13 06:30:35 node1 kernel: [902670.903081] Normal free:40292kB min:3728kB low:4660kB high:5592kB active_anon:4280kB inactive_anon:15772kB
> active_file:199036kB inactive_file:355444kB unevictable:0kB isolated(anon):0kB isolated(file):128kB present:885944kB mlocked:0kB dirty:228kB
> writeback:0kB mapped:24kB shmem:8kB slab_reclaimable:196968kB slab_unreclaimable:15892kB kernel_stack:2608kB pagetables:104kB unstable:0kB
> bounce:108kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
> Mar 13 06:30:35 node1 kernel: [902670.903085] lowmem_reserve[]: 0 0 25321 25321
> Mar 13 06:30:35 node1 kernel: [902670.903091] HighMem free:706596kB min:512kB low:3920kB high:7332kB active_anon:544944kB inactive_anon:142308kB
> active_file:692736kB inactive_file:1140028kB unevictable:16332kB isolated(anon):0kB isolated(file):0kB present:3241132kB mlocked:16332kB dirty:6604kB
> writeback:0kB mapped:94524kB shmem:46144kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:9072kB unstable:0kB bounce:0kB
> writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
> Mar 13 06:30:35 node1 kernel: [902670.903095] lowmem_reserve[]: 0 0 0 0
> Mar 13 06:30:35 node1 kernel: [902670.903097] DMA: 15*4kB 15*8kB 9*16kB 22*32kB 7*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 1*2048kB 0*4096kB = 3524kB
> Mar 13 06:30:35 node1 kernel: [902670.903104] Normal: 3507*4kB 3160*8kB 37*16kB 8*32kB 2*64kB 1*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 40412kB
> Mar 13 06:30:35 node1 kernel: [902670.903110] HighMem: 77*4kB 56*8kB 23*16kB 12*32kB 2061*64kB 2550*128kB 704*256kB 104*512kB 13*1024kB 0*2048kB
> 0*4096kB = 706596kB
> Mar 13 06:30:35 node1 kernel: [902670.903117] 612114 total pagecache pages
> Mar 13 06:30:35 node1 kernel: [902670.903118] 160 pages in swap cache
> Mar 13 06:30:35 node1 kernel: [902670.903120] Swap cache stats: add 21929, delete 21769, find 243894/245286
> Mar 13 06:30:35 node1 kernel: [902670.903121] Free swap = 4880236kB
> Mar 13 06:30:35 node1 kernel: [902670.903122] Total swap = 4883640kB
> Mar 13 06:30:35 node1 kernel: [902670.912442] 1245184 pages RAM
> Mar 13 06:30:35 node1 kernel: [902670.912443] 1017858 pages HighMem
> Mar 13 06:30:35 node1 kernel: [902670.912444] 210949 pages reserved
> Mar 13 06:30:35 node1 kernel: [902670.912445] 327375 pages shared
> Mar 13 06:30:35 node1 kernel: [902670.912446] 642252 pages non-shared
> Mar 13 06:30:35 node1 kernel: [902670.912462] drbd1_receiver: page allocation failure. order:4, mode:0x4020
> Mar 13 06:30:35 node1 kernel: [902670.912465] Pid: 25245, comm: drbd1_receiver Not tainted 2.6.32-5-vserver-686-bigmem #1
> Mar 13 06:30:35 node1 kernel: [902670.912467] Call Trace:
> Mar 13 06:30:35 node1 kernel: [902670.912472] [<c109ad8c>] ? __alloc_pages_nodemask+0x484/0x4d9
> Mar 13 06:30:35 node1 kernel: [902670.912475] [<c109aded>] ? __get_free_pages+0xc/0x17
> Mar 13 06:30:35 node1 kernel: [902670.912478] [<c10bf261>] ? __kmalloc+0x30/0x128
> Mar 13 06:30:35 node1 kernel: [902670.912487] [<c11ee2f4>] ? pskb_expand_head+0x4f/0x157
> Mar 13 06:30:35 node1 kernel: [902670.912490] [<c11ee74b>] ? __pskb_pull_tail+0x40/0x200
> Mar 13 06:30:35 node1 kernel: [902670.912492] [<c11eb077>] ? sock_wfree+0x17/0x4b
> Mar 13 06:30:35 node1 kernel: [902670.912495] [<c11f555e>] ? dev_queue_xmit+0xe4/0x38e
> Mar 13 06:30:35 node1 kernel: [902670.912497] [<c1216cea>] ? ip_finish_output2+0x187/0x1c2
> Mar 13 06:30:35 node1 kernel: [902670.912500] [<c12161ed>] ? ip_local_out+0x15/0x17
> Mar 13 06:30:35 node1 kernel: [902670.912502] [<c12169cc>] ? ip_queue_xmit+0x31d/0x378
> Mar 13 06:30:35 node1 kernel: [902670.912505] [<c1224fdf>] ? tcp_established_options+0x1d/0x8b
> Mar 13 06:30:35 node1 kernel: [902670.912507] [<c1225c52>] ? tcp_transmit_skb+0x595/0x5cc
> Mar 13 06:30:35 node1 kernel: [902670.912510] [<c1227cb5>] ? tcp_write_xmit+0x7a3/0x874
> Mar 13 06:30:35 node1 kernel: [902670.912512] [<c12236e4>] ? tcp_ack+0x1611/0x1802
> Mar 13 06:30:35 node1 kernel: [902670.912514] [<c1224fdf>] ? tcp_established_options+0x1d/0x8b
> Mar 13 06:30:35 node1 kernel: [902670.912516] [<c12252a3>] ? tcp_current_mss+0x38/0x53
> Mar 13 06:30:35 node1 kernel: [902670.912519] [<c1227dcc>] ? __tcp_push_pending_frames+0x1e/0x50
> Mar 13 06:30:35 node1 kernel: [902670.912521] [<c12238f0>] ? tcp_data_snd_check+0x1b/0xd2
> Mar 13 06:30:35 node1 kernel: [902670.912523] [<c1224493>] ? tcp_rcv_established+0x5d6/0x62c
> Mar 13 06:30:35 node1 kernel: [902670.912526] [<c128afe9>] ? schedule_timeout+0x20/0xb0
> Mar 13 06:30:35 node1 kernel: [902670.912529] [<c122a9ae>] ? tcp_v4_do_rcv+0x15f/0x2cf
> Mar 13 06:30:35 node1 kernel: [902670.912531] [<c11eabce>] ? sk_wait_data+0x99/0xa2
> Mar 13 06:30:35 node1 kernel: [902670.912534] [<c121bf64>] ? tcp_prequeue_process+0x57/0x6b
> Mar 13 06:30:35 node1 kernel: [902670.912536] [<c121cf66>] ? tcp_recvmsg+0x441/0x8e5
> Mar 13 06:30:35 node1 kernel: [902670.912538] [<c11e9cdf>] ? sock_common_recvmsg+0x2f/0x45
> Mar 13 06:30:35 node1 kernel: [902670.912541] [<c11e9cdf>] ? sock_common_recvmsg+0x2f/0x45
> Mar 13 06:30:35 node1 kernel: [902670.912544] [<c11e7fc3>] ? __sock_recvmsg+0x50/0xbb
> Mar 13 06:30:35 node1 kernel: [902670.912546] [<c11e855d>] ? sock_recvmsg+0x9d/0xb6
> Mar 13 06:30:35 node1 kernel: [902670.912549] [<c104b08a>] ? autoremove_wake_function+0x0/0x2d
> Mar 13 06:30:35 node1 kernel: [902670.912553] [<c113c658>] ? __make_request+0x364/0x3d9
> Mar 13 06:30:35 node1 kernel: [902670.912558] [<f9067ddf>] ? drbd_recv+0x62/0x106 [drbd]
> Mar 13 06:30:35 node1 kernel: [902670.912563] [<f9067e95>] ? drbd_recv_header+0x12/0xa5 [drbd]
> Mar 13 06:30:35 node1 kernel: [902670.912568] [<f906e669>] ? drbdd_init+0xf3/0x245 [drbd]
> Mar 13 06:30:35 node1 kernel: [902670.912572] [<f9079bf7>] ? drbd_thread_setup+0x1f/0xcc [drbd]
> Mar 13 06:30:35 node1 kernel: [902670.912577] [<f9079bd8>] ? drbd_thread_setup+0x0/0xcc [drbd]
> Mar 13 06:30:35 node1 kernel: [902670.912579] [<c104ae58>] ? kthread+0x61/0x66
> Mar 13 06:30:35 node1 kernel: [902670.912581] [<c104adf7>] ? kthread+0x0/0x66
> Mar 13 06:30:35 node1 kernel: [902670.912584] [<c1008d87>] ? kernel_thread_helper+0x7/0x10
> Mar 13 06:30:35 node1 kernel: [902670.912585] Mem-Info:
> Mar 13 06:30:35 node1 kernel: [902670.912587] DMA per-cpu:
> Mar 13 06:30:35 node1 kernel: [902670.912588] CPU 0: hi: 0, btch: 1 usd: 0
> Mar 13 06:30:35 node1 kernel: [902670.912590] CPU 1: hi: 0, btch: 1 usd: 0
> Mar 13 06:30:35 node1 kernel: [902670.912591] Normal per-cpu:
> Mar 13 06:30:35 node1 kernel: [902670.912592] CPU 0: hi: 186, btch: 31 usd: 173
> Mar 13 06:30:35 node1 kernel: [902670.912594] CPU 1: hi: 186, btch: 31 usd: 154
> Mar 13 06:30:35 node1 kernel: [902670.912595] HighMem per-cpu:
> Mar 13 06:30:35 node1 kernel: [902670.912596] CPU 0: hi: 186, btch: 31 usd: 47
> Mar 13 06:30:35 node1 kernel: [902670.912598] CPU 1: hi: 186, btch: 31 usd: 12
> Mar 13 06:30:35 node1 kernel: [902670.912601] active_anon:137306 inactive_anon:39520 isolated_anon:0
> Mar 13 06:30:35 node1 kernel: [902670.912602] active_file:223625 inactive_file:375450 isolated_file:32
> Mar 13 06:30:35 node1 kernel: [902670.912603] unevictable:4083 dirty:1708 writeback:0 unstable:0
> Mar 13 06:30:35 node1 kernel: [902670.912604] free:187603 slab_reclaimable:50120 slab_unreclaimable:3975
> Mar 13 06:30:35 node1 kernel: [902670.912604] mapped:23637 shmem:11538 pagetables:2294 bounce:27
> Mar 13 06:30:35 node1 kernel: [902670.912609] DMA free:3524kB min:64kB low:80kB high:96kB active_anon:0kB inactive_anon:0kB active_file:2728kB
> inactive_file:6180kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15868kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB
> slab_reclaimable:3512kB slab_unreclaimable:8kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0
> all_unreclaimable? no
> Mar 13 06:30:35 node1 kernel: [902670.912613] lowmem_reserve[]: 0 865 4030 4030
> Mar 13 06:30:35 node1 kernel: [902670.912619] Normal free:40292kB min:3728kB low:4660kB high:5592kB active_anon:4280kB inactive_anon:15772kB
> active_file:199036kB inactive_file:355444kB unevictable:0kB isolated(anon):0kB isolated(file):128kB present:885944kB mlocked:0kB dirty:228kB
> writeback:0kB mapped:24kB shmem:8kB slab_reclaimable:196968kB slab_unreclaimable:15892kB kernel_stack:2608kB pagetables:104kB unstable:0kB
> bounce:108kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
> Mar 13 06:30:35 node1 kernel: [902670.912623] lowmem_reserve[]: 0 0 25321 25321
> Mar 13 06:30:35 node1 kernel: [902670.912629] HighMem free:706596kB min:512kB low:3920kB high:7332kB active_anon:544944kB inactive_anon:142308kB
> active_file:692736kB inactive_file:1140176kB unevictable:16332kB isolated(anon):0kB isolated(file):0kB present:3241132kB mlocked:16332kB dirty:6604kB
> writeback:0kB mapped:94524kB shmem:46144kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:9072kB unstable:0kB bounce:0kB
> writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
> Mar 13 06:30:35 node1 kernel: [902670.912633] lowmem_reserve[]: 0 0 0 0
> Mar 13 06:30:35 node1 kernel: [902670.912636] DMA: 15*4kB 15*8kB 9*16kB 22*32kB 7*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 1*2048kB 0*4096kB = 3524kB
> Mar 13 06:30:35 node1 kernel: [902670.912642] Normal: 3507*4kB 3160*8kB 37*16kB 8*32kB 2*64kB 1*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 40412kB
> Mar 13 06:30:35 node1 kernel: [902670.912648] HighMem: 77*4kB 56*8kB 23*16kB 12*32kB 2061*64kB 2550*128kB 704*256kB 104*512kB 13*1024kB 0*2048kB
> 0*4096kB = 706596kB
> Mar 13 06:30:35 node1 kernel: [902670.912655] 612114 total pagecache pages
> Mar 13 06:30:35 node1 kernel: [902670.912656] 160 pages in swap cache
> Mar 13 06:30:35 node1 kernel: [902670.912658] Swap cache stats: add 21929, delete 21769, find 243894/245286
> Mar 13 06:30:35 node1 kernel: [902670.912659] Free swap = 4880236kB
> Mar 13 06:30:35 node1 kernel: [902670.912660] Total swap = 4883640kB
> Mar 13 06:30:35 node1 kernel: [902670.922012] 1245184 pages RAM
> Mar 13 06:30:35 node1 kernel: [902670.922014] 1017858 pages HighMem
> Mar 13 06:30:35 node1 kernel: [902670.922015] 210949 pages reserved
> Mar 13 06:30:35 node1 kernel: [902670.922016] 327379 pages shared
> Mar 13 06:30:35 node1 kernel: [902670.922017] 642254 pages non-shared
> Mar 13 06:30:35 node1 kernel: [902670.922033] drbd1_receiver: page allocation failure. order:4, mode:0x4020
> Mar 13 06:30:35 node1 kernel: [902670.922036] Pid: 25245, comm: drbd1_receiver Not tainted 2.6.32-5-vserver-686-bigmem #1
> Mar 13 06:30:35 node1 kernel: [902670.922037] Call Trace:
> Mar 13 06:30:35 node1 kernel: [902670.922043] [<c109ad8c>] ? __alloc_pages_nodemask+0x484/0x4d9
> Mar 13 06:30:35 node1 kernel: [902670.922045] [<c109aded>] ? __get_free_pages+0xc/0x17
> Mar 13 06:30:35 node1 kernel: [902670.922048] [<c10bf261>] ? __kmalloc+0x30/0x128
> Mar 13 06:30:35 node1 kernel: [902670.922057] [<c11ee2f4>] ? pskb_expand_head+0x4f/0x157
> Mar 13 06:30:35 node1 kernel: [902670.922060] [<c11ee74b>] ? __pskb_pull_tail+0x40/0x200
> Mar 13 06:30:35 node1 kernel: [902670.922062] [<c11eb077>] ? sock_wfree+0x17/0x4b
> Mar 13 06:30:35 node1 kernel: [902670.922065] [<c11f555e>] ? dev_queue_xmit+0xe4/0x38e
> Mar 13 06:30:35 node1 kernel: [902670.922067] [<c1216cea>] ? ip_finish_output2+0x187/0x1c2
> Mar 13 06:30:35 node1 kernel: [902670.922070] [<c12161ed>] ? ip_local_out+0x15/0x17
> Mar 13 06:30:35 node1 kernel: [902670.922072] [<c12169cc>] ? ip_queue_xmit+0x31d/0x378
> Mar 13 06:30:35 node1 kernel: [902670.922075] [<c1224fdf>] ? tcp_established_options+0x1d/0x8b
> Mar 13 06:30:35 node1 kernel: [902670.922077] [<c1225c52>] ? tcp_transmit_skb+0x595/0x5cc
> Mar 13 06:30:35 node1 kernel: [902670.922079] [<c1227cb5>] ? tcp_write_xmit+0x7a3/0x874
> Mar 13 06:30:35 node1 kernel: [902670.922082] [<c12236e4>] ? tcp_ack+0x1611/0x1802
> Mar 13 06:30:35 node1 kernel: [902670.922084] [<c1224fdf>] ? tcp_established_options+0x1d/0x8b
> Mar 13 06:30:35 node1 kernel: [902670.922086] [<c12252a3>] ? tcp_current_mss+0x38/0x53
> Mar 13 06:30:35 node1 kernel: [902670.922088] [<c1227dcc>] ? __tcp_push_pending_frames+0x1e/0x50
> Mar 13 06:30:35 node1 kernel: [902670.922091] [<c12238f0>] ? tcp_data_snd_check+0x1b/0xd2
> Mar 13 06:30:35 node1 kernel: [902670.922093] [<c1224493>] ? tcp_rcv_established+0x5d6/0x62c
> Mar 13 06:30:35 node1 kernel: [902670.922096] [<c128afe9>] ? schedule_timeout+0x20/0xb0
> Mar 13 06:30:35 node1 kernel: [902670.922098] [<c122a9ae>] ? tcp_v4_do_rcv+0x15f/0x2cf
> Mar 13 06:30:35 node1 kernel: [902670.922101] [<c11eabce>] ? sk_wait_data+0x99/0xa2
> Mar 13 06:30:35 node1 kernel: [902670.922103] [<c121bf64>] ? tcp_prequeue_process+0x57/0x6b
> Mar 13 06:30:35 node1 kernel: [902670.922106] [<c121cf66>] ? tcp_recvmsg+0x441/0x8e5
> Mar 13 06:30:35 node1 kernel: [902670.922108] [<c11e9cdf>] ? sock_common_recvmsg+0x2f/0x45
> Mar 13 06:30:35 node1 kernel: [902670.922110] [<c11e9cdf>] ? sock_common_recvmsg+0x2f/0x45
> Mar 13 06:30:35 node1 kernel: [902670.922114] [<c11e7fc3>] ? __sock_recvmsg+0x50/0xbb
> Mar 13 06:30:35 node1 kernel: [902670.922116] [<c11e855d>] ? sock_recvmsg+0x9d/0xb6
> Mar 13 06:30:35 node1 kernel: [902670.922120] [<c104b08a>] ? autoremove_wake_function+0x0/0x2d
> Mar 13 06:30:35 node1 kernel: [902670.922123] [<c113c658>] ? __make_request+0x364/0x3d9
> Mar 13 06:30:35 node1 kernel: [902670.922129] [<f9067ddf>] ? drbd_recv+0x62/0x106 [drbd]
> Mar 13 06:30:35 node1 kernel: [902670.922134] [<f9067e95>] ? drbd_recv_header+0x12/0xa5 [drbd]
> Mar 13 06:30:35 node1 kernel: [902670.922138] [<f906e669>] ? drbdd_init+0xf3/0x245 [drbd]
> Mar 13 06:30:35 node1 kernel: [902670.922143] [<f9079bf7>] ? drbd_thread_setup+0x1f/0xcc [drbd]
> Mar 13 06:30:35 node1 kernel: [902670.922147] [<f9079bd8>] ? drbd_thread_setup+0x0/0xcc [drbd]
> Mar 13 06:30:35 node1 kernel: [902670.922150] [<c104ae58>] ? kthread+0x61/0x66
> Mar 13 06:30:35 node1 kernel: [902670.922152] [<c104adf7>] ? kthread+0x0/0x66
> Mar 13 06:30:35 node1 kernel: [902670.922155] [<c1008d87>] ? kernel_thread_helper+0x7/0x10
> Mar 13 06:30:35 node1 kernel: [902670.922156] Mem-Info:
> Mar 13 06:30:35 node1 kernel: [902670.922157] DMA per-cpu:
> Mar 13 06:30:35 node1 kernel: [902670.922159] CPU 0: hi: 0, btch: 1 usd: 0
> Mar 13 06:30:35 node1 kernel: [902670.922160] CPU 1: hi: 0, btch: 1 usd: 0
> Mar 13 06:30:35 node1 kernel: [902670.922161] Normal per-cpu:
> Mar 13 06:30:35 node1 kernel: [902670.922163] CPU 0: hi: 186, btch: 31 usd: 173
> Mar 13 06:30:35 node1 kernel: [902670.922164] CPU 1: hi: 186, btch: 31 usd: 154
> Mar 13 06:30:35 node1 kernel: [902670.922166] HighMem per-cpu:
> Mar 13 06:30:35 node1 kernel: [902670.922167] CPU 0: hi: 186, btch: 31 usd: 47
> Mar 13 06:30:35 node1 kernel: [902670.922168] CPU 1: hi: 186, btch: 31 usd: 35
> Mar 13 06:30:35 node1 kernel: [902670.922172] active_anon:137306 inactive_anon:39520 isolated_anon:0
> Mar 13 06:30:35 node1 kernel: [902670.922173] active_file:223625 inactive_file:375450 isolated_file:32
> Mar 13 06:30:35 node1 kernel: [902670.922173] unevictable:4083 dirty:1708 writeback:0 unstable:0
> Mar 13 06:30:35 node1 kernel: [902670.922174] free:187572 slab_reclaimable:50120 slab_unreclaimable:3975
> Mar 13 06:30:35 node1 kernel: [902670.922175] mapped:23637 shmem:11538 pagetables:2294 bounce:27
> Mar 13 06:30:35 node1 kernel: [902670.922180] DMA free:3524kB min:64kB low:80kB high:96kB active_anon:0kB inactive_anon:0kB active_file:2728kB
> inactive_file:6180kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15868kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB
> slab_reclaimable:3512kB slab_unreclaimable:8kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0
> all_unreclaimable? no
> Mar 13 06:30:35 node1 kernel: [902670.922184] lowmem_reserve[]: 0 865 4030 4030
> Mar 13 06:30:35 node1 kernel: [902670.922190] Normal free:40292kB min:3728kB low:4660kB high:5592kB active_anon:4280kB inactive_anon:15772kB
> active_file:199036kB inactive_file:355444kB unevictable:0kB isolated(anon):0kB isolated(file):128kB present:885944kB mlocked:0kB dirty:228kB
> writeback:0kB mapped:24kB shmem:8kB slab_reclaimable:196968kB slab_unreclaimable:15892kB kernel_stack:2608kB pagetables:104kB unstable:0kB
> bounce:108kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
> Mar 13 06:30:35 node1 kernel: [902670.922194] lowmem_reserve[]: 0 0 25321 25321
> Mar 13 06:30:35 node1 kernel: [902670.922199] HighMem free:706472kB min:512kB low:3920kB high:7332kB active_anon:544944kB inactive_anon:142308kB
> active_file:692736kB inactive_file:1140176kB unevictable:16332kB isolated(anon):0kB isolated(file):0kB present:3241132kB mlocked:16332kB dirty:6604kB
> writeback:0kB mapped:94524kB shmem:46144kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:9072kB unstable:0kB bounce:0kB
> writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
> Mar 13 06:30:35 node1 kernel: [902670.922204] lowmem_reserve[]: 0 0 0 0
> Mar 13 06:30:35 node1 kernel: [902670.922206] DMA: 15*4kB 15*8kB 9*16kB 22*32kB 7*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 1*2048kB 0*4096kB = 3524kB
> Mar 13 06:30:35 node1 kernel: [902670.922213] Normal: 3507*4kB 3160*8kB 37*16kB 8*32kB 2*64kB 1*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 40412kB
> Mar 13 06:30:35 node1 kernel: [902670.922219] HighMem: 78*4kB 56*8kB 23*16kB 12*32kB 2059*64kB 2550*128kB 704*256kB 104*512kB 13*1024kB 0*2048kB
> 0*4096kB = 706472kB
> Mar 13 06:30:35 node1 kernel: [902670.922226] 612114 total pagecache pages
> Mar 13 06:30:35 node1 kernel: [902670.922227] 160 pages in swap cache
> Mar 13 06:30:35 node1 kernel: [902670.922228] Swap cache stats: add 21929, delete 21769, find 243894/245286
> Mar 13 06:30:35 node1 kernel: [902670.922230] Free swap = 4880236kB
> Mar 13 06:30:35 node1 kernel: [902670.922231] Total swap = 4883640kB
> Mar 13 06:30:35 node1 kernel: [902670.931613] 1245184 pages RAM
> Mar 13 06:30:35 node1 kernel: [902670.931615] 1017858 pages HighMem
> Mar 13 06:30:35 node1 kernel: [902670.931616] 210949 pages reserved
> Mar 13 06:30:35 node1 kernel: [902670.931617] 327383 pages shared
> Mar 13 06:30:35 node1 kernel: [902670.931618] 642256 pages non-shared
> Mar 13 06:30:35 node1 kernel: [902670.931634] drbd1_receiver: page allocation failure. order:4, mode:0x4020
> Mar 13 06:30:35 node1 kernel: [902670.931636] Pid: 25245, comm: drbd1_receiver Not tainted 2.6.32-5-vserver-686-bigmem #1
> Mar 13 06:30:35 node1 kernel: [902670.931638] Call Trace:
> Mar 13 06:30:35 node1 kernel: [902670.931643] [<c109ad8c>] ? __alloc_pages_nodemask+0x484/0x4d9
> Mar 13 06:30:35 node1 kernel: [902670.931646] [<c109aded>] ? __get_free_pages+0xc/0x17
> Mar 13 06:30:35 node1 kernel: [902670.931649] [<c10bf261>] ? __kmalloc+0x30/0x128
> Mar 13 06:30:35 node1 kernel: [902670.931658] [<c11ee2f4>] ? pskb_expand_head+0x4f/0x157
> Mar 13 06:30:35 node1 kernel: [902670.931660] [<c11ee74b>] ? __pskb_pull_tail+0x40/0x200
> Mar 13 06:30:35 node1 kernel: [902670.931663] [<c11eb077>] ? sock_wfree+0x17/0x4b
> Mar 13 06:30:35 node1 kernel: [902670.931665] [<c11f555e>] ? dev_queue_xmit+0xe4/0x38e
> Mar 13 06:30:35 node1 kernel: [902670.931668] [<c1216cea>] ? ip_finish_output2+0x187/0x1c2
> Mar 13 06:30:35 node1 kernel: [902670.931671] [<c12161ed>] ? ip_local_out+0x15/0x17
> Mar 13 06:30:35 node1 kernel: [902670.931673] [<c12169cc>] ? ip_queue_xmit+0x31d/0x378
> Mar 13 06:30:35 node1 kernel: [902670.931676] [<c1224fdf>] ? tcp_established_options+0x1d/0x8b
> Mar 13 06:30:35 node1 kernel: [902670.931678] [<c1225c52>] ? tcp_transmit_skb+0x595/0x5cc
> Mar 13 06:30:35 node1 kernel: [902670.931681] [<c1227cb5>] ? tcp_write_xmit+0x7a3/0x874
> Mar 13 06:30:35 node1 kernel: [902670.931683] [<c12236e4>] ? tcp_ack+0x1611/0x1802
> Mar 13 06:30:35 node1 kernel: [902670.931685] [<c1224fdf>] ? tcp_established_options+0x1d/0x8b
> Mar 13 06:30:35 node1 kernel: [902670.931687] [<c12252a3>] ? tcp_current_mss+0x38/0x53
> Mar 13 06:30:35 node1 kernel: [902670.931689] [<c1227dcc>] ? __tcp_push_pending_frames+0x1e/0x50
> Mar 13 06:30:35 node1 kernel: [902670.931692] [<c12238f0>] ? tcp_data_snd_check+0x1b/0xd2
> Mar 13 06:30:35 node1 kernel: [902670.931694] [<c1224493>] ? tcp_rcv_established+0x5d6/0x62c
> Mar 13 06:30:35 node1 kernel: [902670.931697] [<c128afe9>] ? schedule_timeout+0x20/0xb0
> Mar 13 06:30:35 node1 kernel: [902670.931700] [<c122a9ae>] ? tcp_v4_do_rcv+0x15f/0x2cf
> Mar 13 06:30:35 node1 kernel: [902670.931702] [<c11eabce>] ? sk_wait_data+0x99/0xa2
> Mar 13 06:30:35 node1 kernel: [902670.931705] [<c121bf64>] ? tcp_prequeue_process+0x57/0x6b
> Mar 13 06:30:35 node1 kernel: [902670.931707] [<c121cf66>] ? tcp_recvmsg+0x441/0x8e5
> Mar 13 06:30:35 node1 kernel: [902670.931709] [<c11e9cdf>] ? sock_common_recvmsg+0x2f/0x45
> Mar 13 06:30:35 node1 kernel: [902670.931712] [<c11e9cdf>] ? sock_common_recvmsg+0x2f/0x45
> Mar 13 06:30:35 node1 kernel: [902670.931715] [<c11e7fc3>] ? __sock_recvmsg+0x50/0xbb
> Mar 13 06:30:35 node1 kernel: [902670.931717] [<c11e855d>] ? sock_recvmsg+0x9d/0xb6
> Mar 13 06:30:35 node1 kernel: [902670.931720] [<c104b08a>] ? autoremove_wake_function+0x0/0x2d
> Mar 13 06:30:35 node1 kernel: [902670.931724] [<c113c658>] ? __make_request+0x364/0x3d9
> Mar 13 06:30:35 node1 kernel: [902670.931729] [<f9067ddf>] ? drbd_recv+0x62/0x106 [drbd]
> Mar 13 06:30:35 node1 kernel: [902670.931734] [<f9067e95>] ? drbd_recv_header+0x12/0xa5 [drbd]
> Mar 13 06:30:35 node1 kernel: [902670.931739] [<f906e669>] ? drbdd_init+0xf3/0x245 [drbd]
> Mar 13 06:30:35 node1 kernel: [902670.931743] [<f9079bf7>] ? drbd_thread_setup+0x1f/0xcc [drbd]
> Mar 13 06:30:35 node1 kernel: [902670.931748] [<f9079bd8>] ? drbd_thread_setup+0x0/0xcc [drbd]
> Mar 13 06:30:35 node1 kernel: [902670.931750] [<c104ae58>] ? kthread+0x61/0x66
> Mar 13 06:30:35 node1 kernel: [902670.931752] [<c104adf7>] ? kthread+0x0/0x66
> Mar 13 06:30:35 node1 kernel: [902670.931755] [<c1008d87>] ? kernel_thread_helper+0x7/0x10
> Mar 13 06:30:35 node1 kernel: [902670.931756] Mem-Info:
> Mar 13 06:30:35 node1 kernel: [902670.931757] DMA per-cpu:
> Mar 13 06:30:35 node1 kernel: [902670.931759] CPU 0: hi: 0, btch: 1 usd: 0
> Mar 13 06:30:35 node1 kernel: [902670.931760] CPU 1: hi: 0, btch: 1 usd: 0
> Mar 13 06:30:35 node1 kernel: [902670.931762] Normal per-cpu:
> Mar 13 06:30:35 node1 kernel: [902670.931763] CPU 0: hi: 186, btch: 31 usd: 173
> Mar 13 06:30:35 node1 kernel: [902670.931764] CPU 1: hi: 186, btch: 31 usd: 154
> Mar 13 06:30:35 node1 kernel: [902670.931766] HighMem per-cpu:
> Mar 13 06:30:35 node1 kernel: [902670.931767] CPU 0: hi: 186, btch: 31 usd: 47
> Mar 13 06:30:35 node1 kernel: [902670.931769] CPU 1: hi: 186, btch: 31 usd: 27
> Mar 13 06:30:35 node1 kernel: [902670.931772] active_anon:137306 inactive_anon:39520 isolated_anon:0
> Mar 13 06:30:35 node1 kernel: [902670.931773] active_file:223625 inactive_file:375450 isolated_file:32
> Mar 13 06:30:35 node1 kernel: [902670.931774] unevictable:4083 dirty:1708 writeback:0 unstable:0
> Mar 13 06:30:35 node1 kernel: [902670.931774] free:187572 slab_reclaimable:50120 slab_unreclaimable:3975
> Mar 13 06:30:35 node1 kernel: [902670.931775] mapped:23637 shmem:11538 pagetables:2294 bounce:27
> Mar 13 06:30:35 node1 kernel: [902670.931780] DMA free:3524kB min:64kB low:80kB high:96kB active_anon:0kB inactive_anon:0kB active_file:2728kB
> inactive_file:6180kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15868kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB
> slab_reclaimable:3512kB slab_unreclaimable:8kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0
> all_unreclaimable? no
> Mar 13 06:30:35 node1 kernel: [902670.931784] lowmem_reserve[]: 0 865 4030 4030
> Mar 13 06:30:35 node1 kernel: [902670.931790] Normal free:40292kB min:3728kB low:4660kB high:5592kB active_anon:4280kB inactive_anon:15772kB
> active_file:199036kB inactive_file:355444kB unevictable:0kB isolated(anon):0kB isolated(file):128kB present:885944kB mlocked:0kB dirty:228kB
> writeback:0kB mapped:24kB shmem:8kB slab_reclaimable:196968kB slab_unreclaimable:15892kB kernel_stack:2608kB pagetables:104kB unstable:0kB
> bounce:108kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
> Mar 13 06:30:35 node1 kernel: [902670.931794] lowmem_reserve[]: 0 0 25321 25321
> Mar 13 06:30:35 node1 kernel: [902670.931799] HighMem free:706472kB min:512kB low:3920kB high:7332kB active_anon:544944kB inactive_anon:142308kB
> active_file:692736kB inactive_file:1140176kB unevictable:16332kB isolated(anon):0kB isolated(file):0kB present:3241132kB mlocked:16332kB dirty:6604kB
> writeback:0kB mapped:94524kB shmem:46144kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:9072kB unstable:0kB bounce:0kB
> writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
> Mar 13 06:30:35 node1 kernel: [902670.931804] lowmem_reserve[]: 0 0 0 0
> Mar 13 06:30:35 node1 kernel: [902670.931806] DMA: 15*4kB 15*8kB 9*16kB 22*32kB 7*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 1*2048kB 0*4096kB = 3524kB
> Mar 13 06:30:35 node1 kernel: [902670.931813] Normal: 3507*4kB 3160*8kB 37*16kB 8*32kB 2*64kB 1*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 40412kB
> Mar 13 06:30:35 node1 kernel: [902670.931819] HighMem: 78*4kB 56*8kB 23*16kB 12*32kB 2059*64kB 2550*128kB 704*256kB 104*512kB 13*1024kB 0*2048kB
> 0*4096kB = 706472kB
> Mar 13 06:30:35 node1 kernel: [902670.931826] 612114 total pagecache pages
> Mar 13 06:30:35 node1 kernel: [902670.931827] 160 pages in swap cache
> Mar 13 06:30:35 node1 kernel: [902670.931828] Swap cache stats: add 21929, delete 21769, find 243894/245286
> Mar 13 06:30:35 node1 kernel: [902670.931830] Free swap = 4880236kB
> Mar 13 06:30:35 node1 kernel: [902670.931831] Total swap = 4883640kB
> Mar 13 06:30:35 node1 kernel: [902670.941170] 1245184 pages RAM
> Mar 13 06:30:35 node1 kernel: [902670.941171] 1017858 pages HighMem
> Mar 13 06:30:35 node1 kernel: [902670.941172] 210949 pages reserved
> Mar 13 06:30:35 node1 kernel: [902670.941173] 327394 pages shared
> Mar 13 06:30:35 node1 kernel: [902670.941174] 642258 pages non-shared
> Mar 13 06:30:35 node1 kernel: [902670.941986] drbd1_receiver: page allocation failure. order:4, mode:0x4020
> Mar 13 06:30:35 node1 kernel: [902670.941989] Pid: 25245, comm: drbd1_receiver Not tainted 2.6.32-5-vserver-686-bigmem #1
> Mar 13 06:30:35 node1 kernel: [902670.941990] Call Trace:
> Mar 13 06:30:35 node1 kernel: [902670.941997] [<c109ad8c>] ? __alloc_pages_nodemask+0x484/0x4d9
> Mar 13 06:30:35 node1 kernel: [902670.941999] [<c109aded>] ? __get_free_pages+0xc/0x17
> Mar 13 06:30:35 node1 kernel: [902670.942002] [<c10bf261>] ? __kmalloc+0x30/0x128
> Mar 13 06:30:35 node1 kernel: [902670.942013] [<c11ee2f4>] ? pskb_expand_head+0x4f/0x157
> Mar 13 06:30:35 node1 kernel: [902670.942015] [<c11ee74b>] ? __pskb_pull_tail+0x40/0x200
> Mar 13 06:30:35 node1 kernel: [902670.942018] [<c11f555e>] ? dev_queue_xmit+0xe4/0x38e
> Mar 13 06:30:35 node1 kernel: [902670.942021] [<c1216cea>] ? ip_finish_output2+0x187/0x1c2
> Mar 13 06:30:35 node1 kernel: [902670.942024] [<c12161ed>] ? ip_local_out+0x15/0x17
> Mar 13 06:30:35 node1 kernel: [902670.942026] [<c12169cc>] ? ip_queue_xmit+0x31d/0x378
> Mar 13 06:30:35 node1 kernel: [902670.942033] [<f81b7696>] ? scsi_init_sgtable+0x2d/0x46 [scsi_mod]
> Mar 13 06:30:35 node1 kernel: [902670.942037] [<c1225c52>] ? tcp_transmit_skb+0x595/0x5cc
> Mar 13 06:30:35 node1 kernel: [902670.942039] [<c1227cb5>] ? tcp_write_xmit+0x7a3/0x874
> Mar 13 06:30:35 node1 kernel: [902670.942041] [<c12236e4>] ? tcp_ack+0x1611/0x1802
> Mar 13 06:30:35 node1 kernel: [902670.942044] [<c1224fdf>] ? tcp_established_options+0x1d/0x8b
> Mar 13 06:30:35 node1 kernel: [902670.942046] [<c12252a3>] ? tcp_current_mss+0x38/0x53
> Mar 13 06:30:35 node1 kernel: [902670.942049] [<c1227dcc>] ? __tcp_push_pending_frames+0x1e/0x50
> Mar 13 06:30:35 node1 kernel: [902670.942051] [<c12238f0>] ? tcp_data_snd_check+0x1b/0xd2
> Mar 13 06:30:35 node1 kernel: [902670.942053] [<c1224493>] ? tcp_rcv_established+0x5d6/0x62c
> Mar 13 06:30:35 node1 kernel: [902670.942056] [<c122a9ae>] ? tcp_v4_do_rcv+0x15f/0x2cf
> Mar 13 06:30:35 node1 kernel: [902670.942058] [<c11f2e36>] ? net_tx_action+0xa9/0xf9
> Mar 13 06:30:35 node1 kernel: [902670.942060] [<c11ea5ff>] ? release_sock+0x38/0x89
> Mar 13 06:30:35 node1 kernel: [902670.942063] [<c11eab8a>] ? sk_wait_data+0x55/0xa2
> Mar 13 06:30:35 node1 kernel: [902670.942066] [<c104b08a>] ? autoremove_wake_function+0x0/0x2d
> Mar 13 06:30:35 node1 kernel: [902670.942068] [<c121cf09>] ? tcp_recvmsg+0x3e4/0x8e5
> Mar 13 06:30:35 node1 kernel: [902670.942071] [<c11e9cdf>] ? sock_common_recvmsg+0x2f/0x45
> Mar 13 06:30:35 node1 kernel: [902670.942073] [<c11e9cdf>] ? sock_common_recvmsg+0x2f/0x45
> Mar 13 06:30:35 node1 kernel: [902670.942076] [<c11e7fc3>] ? __sock_recvmsg+0x50/0xbb
> Mar 13 06:30:35 node1 kernel: [902670.942078] [<c11e855d>] ? sock_recvmsg+0x9d/0xb6
> Mar 13 06:30:35 node1 kernel: [902670.942081] [<c104b08a>] ? autoremove_wake_function+0x0/0x2d
> Mar 13 06:30:35 node1 kernel: [902670.942084] [<c113c658>] ? __make_request+0x364/0x3d9
> Mar 13 06:30:35 node1 kernel: [902670.942089] [<f9067ddf>] ? drbd_recv+0x62/0x106 [drbd]
> Mar 13 06:30:35 node1 kernel: [902670.942094] [<f9067e95>] ? drbd_recv_header+0x12/0xa5 [drbd]
> Mar 13 06:30:35 node1 kernel: [902670.942099] [<f906e669>] ? drbdd_init+0xf3/0x245 [drbd]
> Mar 13 06:30:35 node1 kernel: [902670.942103] [<f9079bf7>] ? drbd_thread_setup+0x1f/0xcc [drbd]
> Mar 13 06:30:35 node1 kernel: [902670.942108] [<f9079bd8>] ? drbd_thread_setup+0x0/0xcc [drbd]
> Mar 13 06:30:35 node1 kernel: [902670.942110] [<c104ae58>] ? kthread+0x61/0x66
> Mar 13 06:30:35 node1 kernel: [902670.942112] [<c104adf7>] ? kthread+0x0/0x66
> Mar 13 06:30:35 node1 kernel: [902670.942115] [<c1008d87>] ? kernel_thread_helper+0x7/0x10
> Mar 13 06:30:35 node1 kernel: [902670.942117] Mem-Info:
> Mar 13 06:30:35 node1 kernel: [902670.942118] DMA per-cpu:
> Mar 13 06:30:35 node1 kernel: [902670.942119] CPU 0: hi: 0, btch: 1 usd: 0
> Mar 13 06:30:35 node1 kernel: [902670.942121] CPU 1: hi: 0, btch: 1 usd: 0
> Mar 13 06:30:35 node1 kernel: [902670.942122] Normal per-cpu:
> Mar 13 06:30:35 node1 kernel: [902670.942124] CPU 0: hi: 186, btch: 31 usd: 180
> Mar 13 06:30:35 node1 kernel: [902670.942125] CPU 1: hi: 186, btch: 31 usd: 154
> Mar 13 06:30:35 node1 kernel: [902670.942126] HighMem per-cpu:
> Mar 13 06:30:35 node1 kernel: [902670.942128] CPU 0: hi: 186, btch: 31 usd: 47
> Mar 13 06:30:35 node1 kernel: [902670.942129] CPU 1: hi: 186, btch: 31 usd: 19
> Mar 13 06:30:35 node1 kernel: [902670.942133] active_anon:137306 inactive_anon:39520 isolated_anon:0
> Mar 13 06:30:35 node1 kernel: [902670.942133] active_file:223625 inactive_file:375450 isolated_file:32
> Mar 13 06:30:35 node1 kernel: [902670.942134] unevictable:4083 dirty:1708 writeback:0 unstable:0
> Mar 13 06:30:35 node1 kernel: [902670.942135] free:187574 slab_reclaimable:50120 slab_unreclaimable:3993
> Mar 13 06:30:35 node1 kernel: [902670.942136] mapped:23637 shmem:11538 pagetables:2294 bounce:27
> Mar 13 06:30:35 node1 kernel: [902670.942141] DMA free:3524kB min:64kB low:80kB high:96kB active_anon:0kB inactive_anon:0kB active_file:2728kB
> inactive_file:6180kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15868kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB
> slab_reclaimable:3512kB slab_unreclaimable:8kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0
> all_unreclaimable? no
> Mar 13 06:30:35 node1 kernel: [902670.942144] lowmem_reserve[]: 0 865 4030 4030
> Mar 13 06:30:35 node1 kernel: [902670.942150] Normal free:40300kB min:3728kB low:4660kB high:5592kB active_anon:4280kB inactive_anon:15772kB
> active_file:199036kB inactive_file:355444kB unevictable:0kB isolated(anon):0kB isolated(file):128kB present:885944kB mlocked:0kB dirty:228kB
> writeback:0kB mapped:24kB shmem:8kB slab_reclaimable:196968kB slab_unreclaimable:15964kB kernel_stack:2608kB pagetables:104kB unstable:0kB
> bounce:108kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
> Mar 13 06:30:35 node1 kernel: [902670.942155] lowmem_reserve[]: 0 0 25321 25321
> Mar 13 06:30:35 node1 kernel: [902670.942160] HighMem free:706472kB min:512kB low:3920kB high:7332kB active_anon:544944kB inactive_anon:142308kB
> active_file:692736kB inactive_file:1140176kB unevictable:16332kB isolated(anon):0kB isolated(file):0kB present:3241132kB mlocked:16332kB dirty:6604kB
> writeback:0kB mapped:94524kB shmem:46144kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:9072kB unstable:0kB bounce:0kB
> writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
> Mar 13 06:30:35 node1 kernel: [902670.942165] lowmem_reserve[]: 0 0 0 0
> Mar 13 06:30:35 node1 kernel: [902670.942168] DMA: 15*4kB 15*8kB 9*16kB 22*32kB 7*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 1*2048kB 0*4096kB = 3524kB
> Mar 13 06:30:35 node1 kernel: [902670.942174] Normal: 3507*4kB 3159*8kB 39*16kB 8*32kB 1*64kB 1*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 40372kB
> Mar 13 06:30:35 node1 kernel: [902670.942180] HighMem: 78*4kB 56*8kB 23*16kB 12*32kB 2059*64kB 2550*128kB 704*256kB 104*512kB 13*1024kB 0*2048kB
> 0*4096kB = 706472kB
> Mar 13 06:30:35 node1 kernel: [902670.942187] 612114 total pagecache pages
> Mar 13 06:30:35 node1 kernel: [902670.942188] 160 pages in swap cache
> Mar 13 06:30:35 node1 kernel: [902670.942190] Swap cache stats: add 21929, delete 21769, find 243894/245286
> Mar 13 06:30:35 node1 kernel: [902670.942191] Free swap = 4880236kB
> Mar 13 06:30:35 node1 kernel: [902670.942192] Total swap = 4883640kB
> Mar 13 06:30:35 node1 kernel: [902670.951693] 1245184 pages RAM
> Mar 13 06:30:35 node1 kernel: [902670.951696] 1017858 pages HighMem
> Mar 13 06:30:35 node1 kernel: [902670.951697] 210949 pages reserved
> Mar 13 06:30:35 node1 kernel: [902670.951698] 327195 pages shared
> Mar 13 06:30:35 node1 kernel: [902670.951699] 642296 pages non-shared
> Mar 13 06:30:35 node1 kernel: [902670.951716] drbd1_receiver: page allocation failure. order:4, mode:0x4020
> Mar 13 06:30:35 node1 kernel: [902670.951718] Pid: 25245, comm: drbd1_receiver Not tainted 2.6.32-5-vserver-686-bigmem #1
> Mar 13 06:30:35 node1 kernel: [902670.951720] Call Trace:
> Mar 13 06:30:35 node1 kernel: [902670.951725] [<c109ad8c>] ? __alloc_pages_nodemask+0x484/0x4d9
> Mar 13 06:30:35 node1 kernel: [902670.951728] [<c109aded>] ? __get_free_pages+0xc/0x17
> Mar 13 06:30:35 node1 kernel: [902670.951731] [<c10bf261>] ? __kmalloc+0x30/0x128
> Mar 13 06:30:35 node1 kernel: [902670.951740] [<c11ee2f4>] ? pskb_expand_head+0x4f/0x157
> Mar 13 06:30:35 node1 kernel: [902670.951742] [<c11ee74b>] ? __pskb_pull_tail+0x40/0x200
> Mar 13 06:30:35 node1 kernel: [902670.951744] [<c11eb077>] ? sock_wfree+0x17/0x4b
> Mar 13 06:30:35 node1 kernel: [902670.951747] [<c11f555e>] ? dev_queue_xmit+0xe4/0x38e
> Mar 13 06:30:35 node1 kernel: [902670.951750] [<c1216cea>] ? ip_finish_output2+0x187/0x1c2
> Mar 13 06:30:35 node1 kernel: [902670.951752] [<c12161ed>] ? ip_local_out+0x15/0x17
> Mar 13 06:30:35 node1 kernel: [902670.951754] [<c12169cc>] ? ip_queue_xmit+0x31d/0x378
> Mar 13 06:30:35 node1 kernel: [902670.951762] [<f81b7696>] ? scsi_init_sgtable+0x2d/0x46 [scsi_mod]
> Mar 13 06:30:35 node1 kernel: [902670.951765] [<c1225c52>] ? tcp_transmit_skb+0x595/0x5cc
> Mar 13 06:30:35 node1 kernel: [902670.951767] [<c1227cb5>] ? tcp_write_xmit+0x7a3/0x874
> Mar 13 06:30:35 node1 kernel: [902670.951769] [<c12236e4>] ? tcp_ack+0x1611/0x1802
> Mar 13 06:30:35 node1 kernel: [902670.951771] [<c122448c>] ? tcp_rcv_established+0x5cf/0x62c
> Mar 13 06:30:35 node1 kernel: [902670.951774] [<c1224fdf>] ? tcp_established_options+0x1d/0x8b
> Mar 13 06:30:35 node1 kernel: [902670.951776] [<c12252a3>] ? tcp_current_mss+0x38/0x53
> Mar 13 06:30:35 node1 kernel: [902670.951778] [<c1227dcc>] ? __tcp_push_pending_frames+0x1e/0x50
> Mar 13 06:30:35 node1 kernel: [902670.951780] [<c12238f0>] ? tcp_data_snd_check+0x1b/0xd2
> Mar 13 06:30:35 node1 kernel: [902670.951782] [<c1224493>] ? tcp_rcv_established+0x5d6/0x62c
> Mar 13 06:30:35 node1 kernel: [902670.951785] [<c122a9ae>] ? tcp_v4_do_rcv+0x15f/0x2cf
> Mar 13 06:30:35 node1 kernel: [902670.951787] [<c11f2e36>] ? net_tx_action+0xa9/0xf9
> Mar 13 06:30:35 node1 kernel: [902670.951789] [<c11ea5ff>] ? release_sock+0x38/0x89
> Mar 13 06:30:35 node1 kernel: [902670.951791] [<c11eab8a>] ? sk_wait_data+0x55/0xa2
> Mar 13 06:30:35 node1 kernel: [902670.951794] [<c104b08a>] ? autoremove_wake_function+0x0/0x2d
> Mar 13 06:30:35 node1 kernel: [902670.951797] [<c121cf09>] ? tcp_recvmsg+0x3e4/0x8e5
> Mar 13 06:30:35 node1 kernel: [902670.951799] [<c11e9cdf>] ? sock_common_recvmsg+0x2f/0x45
> Mar 13 06:30:35 node1 kernel: [902670.951801] [<c11e9cdf>] ? sock_common_recvmsg+0x2f/0x45
> Mar 13 06:30:35 node1 kernel: [902670.951805] [<c11e7fc3>] ? __sock_recvmsg+0x50/0xbb
> Mar 13 06:30:35 node1 kernel: [902670.951807] [<c11e855d>] ? sock_recvmsg+0x9d/0xb6
> Mar 13 06:30:35 node1 kernel: [902670.951809] [<c104b08a>] ? autoremove_wake_function+0x0/0x2d
> Mar 13 06:30:35 node1 kernel: [902670.951812] [<c113c658>] ? __make_request+0x364/0x3d9
> Mar 13 06:30:35 node1 kernel: [902670.951818] [<f9067ddf>] ? drbd_recv+0x62/0x106 [drbd]
> Mar 13 06:30:35 node1 kernel: [902670.951822] [<f9067e95>] ? drbd_recv_header+0x12/0xa5 [drbd]
> Mar 13 06:30:35 node1 kernel: [902670.951827] [<f906e669>] ? drbdd_init+0xf3/0x245 [drbd]
> Mar 13 06:30:35 node1 kernel: [902670.951831] [<f9079bf7>] ? drbd_thread_setup+0x1f/0xcc [drbd]
> Mar 13 06:30:35 node1 kernel: [902670.951836] [<f9079bd8>] ? drbd_thread_setup+0x0/0xcc [drbd]
> Mar 13 06:30:35 node1 kernel: [902670.951838] [<c104ae58>] ? kthread+0x61/0x66
> Mar 13 06:30:35 node1 kernel: [902670.951840] [<c104adf7>] ? kthread+0x0/0x66
> Mar 13 06:30:35 node1 kernel: [902670.951843] [<c1008d87>] ? kernel_thread_helper+0x7/0x10
> Mar 13 06:30:35 node1 kernel: [902670.951844] Mem-Info:
> Mar 13 06:30:35 node1 kernel: [902670.951845] DMA per-cpu:
> Mar 13 06:30:35 node1 kernel: [902670.951847] CPU 0: hi: 0, btch: 1 usd: 0
> Mar 13 06:30:35 node1 kernel: [902670.951848] CPU 1: hi: 0, btch: 1 usd: 0
> Mar 13 06:30:35 node1 kernel: [902670.951850] Normal per-cpu:
> Mar 13 06:30:35 node1 kernel: [902670.951851] CPU 0: hi: 186, btch: 31 usd: 180
> Mar 13 06:30:35 node1 kernel: [902670.951853] CPU 1: hi: 186, btch: 31 usd: 138
> Mar 13 06:30:35 node1 kernel: [902670.951855] HighMem per-cpu:
> Mar 13 06:30:35 node1 kernel: [902670.951856] CPU 0: hi: 186, btch: 31 usd: 47
> Mar 13 06:30:35 node1 kernel: [902670.951858] CPU 1: hi: 186, btch: 31 usd: 10
> Mar 13 06:30:35 node1 kernel: [902670.951861] active_anon:137306 inactive_anon:39520 isolated_anon:0
> Mar 13 06:30:35 node1 kernel: [902670.951862] active_file:223625 inactive_file:375487 isolated_file:32
> Mar 13 06:30:35 node1 kernel: [902670.951863] unevictable:4083 dirty:1745 writeback:0 unstable:0
> Mar 13 06:30:35 node1 kernel: [902670.951863] free:187543 slab_reclaimable:50120 slab_unreclaimable:3993
> Mar 13 06:30:35 node1 kernel: [902670.951864] mapped:23637 shmem:11538 pagetables:2294 bounce:52
> Mar 13 06:30:35 node1 kernel: [902670.951869] DMA free:3524kB min:64kB low:80kB high:96kB active_anon:0kB inactive_anon:0kB active_file:2728kB
> inactive_file:6180kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15868kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB
> slab_reclaimable:3512kB slab_unreclaimable:8kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0
> all_unreclaimable? no
> Mar 13 06:30:35 node1 kernel: [902670.951873] lowmem_reserve[]: 0 865 4030 4030
> Mar 13 06:30:35 node1 kernel: [902670.951879] Normal free:40300kB min:3728kB low:4660kB high:5592kB active_anon:4280kB inactive_anon:15772kB
> active_file:199036kB inactive_file:355444kB unevictable:0kB isolated(anon):0kB isolated(file):128kB present:885944kB mlocked:0kB dirty:228kB
> writeback:0kB mapped:24kB shmem:8kB slab_reclaimable:196968kB slab_unreclaimable:15964kB kernel_stack:2608kB pagetables:104kB unstable:0kB
> bounce:208kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
> Mar 13 06:30:35 node1 kernel: [902670.951883] lowmem_reserve[]: 0 0 25321 25321
> Mar 13 06:30:35 node1 kernel: [902670.951889] HighMem free:706348kB min:512kB low:3920kB high:7332kB active_anon:544944kB inactive_anon:142308kB
> active_file:692736kB inactive_file:1140324kB unevictable:16332kB isolated(anon):0kB isolated(file):0kB present:3241132kB mlocked:16332kB dirty:6752kB
> writeback:0kB mapped:94524kB shmem:46144kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:9072kB unstable:0kB bounce:0kB
> writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
> Mar 13 06:30:35 node1 kernel: [902670.951893] lowmem_reserve[]: 0 0 0 0
> Mar 13 06:30:35 node1 kernel: [902670.951896] DMA: 15*4kB 15*8kB 9*16kB 22*32kB 7*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 1*2048kB 0*4096kB = 3524kB
> Mar 13 06:30:35 node1 kernel: [902670.951902] Normal: 3507*4kB 3158*8kB 39*16kB 8*32kB 1*64kB 1*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 40364kB
> Mar 13 06:30:35 node1 kernel: [902670.951908] HighMem: 77*4kB 57*8kB 23*16kB 12*32kB 2057*64kB 2550*128kB 704*256kB 104*512kB 13*1024kB 0*2048kB
> 0*4096kB = 706348kB
> Mar 13 06:30:35 node1 kernel: [902670.951915] 612151 total pagecache pages
> Mar 13 06:30:35 node1 kernel: [902670.951917] 160 pages in swap cache
> Mar 13 06:30:35 node1 kernel: [902670.951918] Swap cache stats: add 21929, delete 21769, find 243894/245286
> Mar 13 06:30:35 node1 kernel: [902670.951920] Free swap = 4880236kB
> Mar 13 06:30:35 node1 kernel: [902670.951921] Total swap = 4883640kB
> Mar 13 06:30:35 node1 kernel: [902670.961279] 1245184 pages RAM
> Mar 13 06:30:35 node1 kernel: [902670.961280] 1017858 pages HighMem
> Mar 13 06:30:35 node1 kernel: [902670.961281] 210949 pages reserved
> Mar 13 06:30:35 node1 kernel: [902670.961282] 327205 pages shared
> Mar 13 06:30:35 node1 kernel: [902670.961283] 642316 pages non-shared
> Mar 13 06:30:35 node1 kernel: [902670.961299] drbd1_receiver: page allocation failure. order:4, mode:0x4020
> Mar 13 06:30:35 node1 kernel: [902670.961302] Pid: 25245, comm: drbd1_receiver Not tainted 2.6.32-5-vserver-686-bigmem #1
> Mar 13 06:30:35 node1 kernel: [902670.961304] Call Trace:
> Mar 13 06:30:35 node1 kernel: [902670.961309] [<c109ad8c>] ? __alloc_pages_nodemask+0x484/0x4d9
> Mar 13 06:30:35 node1 kernel: [902670.961312] [<c109aded>] ? __get_free_pages+0xc/0x17
> Mar 13 06:30:35 node1 kernel: [902670.961315] [<c10bf261>] ? __kmalloc+0x30/0x128
> Mar 13 06:30:35 node1 kernel: [902670.961324] [<c11ee2f4>] ? pskb_expand_head+0x4f/0x157
> Mar 13 06:30:35 node1 kernel: [902670.961327] [<c11ee74b>] ? __pskb_pull_tail+0x40/0x200
> Mar 13 06:30:35 node1 kernel: [902670.961329] [<c11eb077>] ? sock_wfree+0x17/0x4b
> Mar 13 06:30:35 node1 kernel: [902670.961332] [<c11f555e>] ? dev_queue_xmit+0xe4/0x38e
> Mar 13 06:30:35 node1 kernel: [902670.961334] [<c1216cea>] ? ip_finish_output2+0x187/0x1c2
> Mar 13 06:30:35 node1 kernel: [902670.961337] [<c12161ed>] ? ip_local_out+0x15/0x17
> Mar 13 06:30:35 node1 kernel: [902670.961339] [<c12169cc>] ? ip_queue_xmit+0x31d/0x378
> Mar 13 06:30:35 node1 kernel: [902670.961346] [<f81b7696>] ? scsi_init_sgtable+0x2d/0x46 [scsi_mod]
> Mar 13 06:30:35 node1 kernel: [902670.961349] [<c1225c52>] ? tcp_transmit_skb+0x595/0x5cc
> Mar 13 06:30:35 node1 kernel: [902670.961351] [<c1227cb5>] ? tcp_write_xmit+0x7a3/0x874
> Mar 13 06:30:35 node1 kernel: [902670.961353] [<c12236e4>] ? tcp_ack+0x1611/0x1802
> Mar 13 06:30:35 node1 kernel: [902670.961356] [<c1224fdf>] ? tcp_established_options+0x1d/0x8b
> Mar 13 06:30:35 node1 kernel: [902670.961358] [<c12252a3>] ? tcp_current_mss+0x38/0x53
> Mar 13 06:30:35 node1 kernel: [902670.961360] [<c1227dcc>] ? __tcp_push_pending_frames+0x1e/0x50
> Mar 13 06:30:35 node1 kernel: [902670.961362] [<c12238f0>] ? tcp_data_snd_check+0x1b/0xd2
> Mar 13 06:30:35 node1 kernel: [902670.961364] [<c1224493>] ? tcp_rcv_established+0x5d6/0x62c
> Mar 13 06:30:35 node1 kernel: [902670.961367] [<c122a9ae>] ? tcp_v4_do_rcv+0x15f/0x2cf
> Mar 13 06:30:35 node1 kernel: [902670.961369] [<c11f2e36>] ? net_tx_action+0xa9/0xf9
> Mar 13 06:30:35 node1 kernel: [902670.961371] [<c11ea5ff>] ? release_sock+0x38/0x89
> Mar 13 06:30:35 node1 kernel: [902670.961373] [<c11eab8a>] ? sk_wait_data+0x55/0xa2
> Mar 13 06:30:35 node1 kernel: [902670.961376] [<c104b08a>] ? autoremove_wake_function+0x0/0x2d
> Mar 13 06:30:35 node1 kernel: [902670.961379] [<c121cf09>] ? tcp_recvmsg+0x3e4/0x8e5
> Mar 13 06:30:35 node1 kernel: [902670.961381] [<c11e9cdf>] ? sock_common_recvmsg+0x2f/0x45
> Mar 13 06:30:35 node1 kernel: [902670.961383] [<c11e9cdf>] ? sock_common_recvmsg+0x2f/0x45
> Mar 13 06:30:35 node1 kernel: [902670.961387] [<c11e7fc3>] ? __sock_recvmsg+0x50/0xbb
> Mar 13 06:30:35 node1 kernel: [902670.961389] [<c11e855d>] ? sock_recvmsg+0x9d/0xb6
> Mar 13 06:30:35 node1 kernel: [902670.961397] [<c104b08a>] ? autoremove_wake_function+0x0/0x2d
> Mar 13 06:30:35 node1 kernel: [902670.961400] [<c113c658>] ? __make_request+0x364/0x3d9
> Mar 13 06:30:35 node1 kernel: [902670.961406] [<f9067ddf>] ? drbd_recv+0x62/0x106 [drbd]
> Mar 13 06:30:35 node1 kernel: [902670.961410] [<f9067e95>] ? drbd_recv_header+0x12/0xa5 [drbd]
> Mar 13 06:30:35 node1 kernel: [902670.961415] [<f906e669>] ? drbdd_init+0xf3/0x245 [drbd]
> Mar 13 06:30:35 node1 kernel: [902670.961419] [<f9079bf7>] ? drbd_thread_setup+0x1f/0xcc [drbd]
> Mar 13 06:30:35 node1 kernel: [902670.961424] [<f9079bd8>] ? drbd_thread_setup+0x0/0xcc [drbd]
> Mar 13 06:30:35 node1 kernel: [902670.961426] [<c104ae58>] ? kthread+0x61/0x66
> Mar 13 06:30:35 node1 kernel: [902670.961428] [<c104adf7>] ? kthread+0x0/0x66
> Mar 13 06:30:35 node1 kernel: [902670.961431] [<c1008d87>] ? kernel_thread_helper+0x7/0x10
> Mar 13 06:30:35 node1 kernel: [902670.961432] Mem-Info:
> Mar 13 06:30:35 node1 kernel: [902670.961433] DMA per-cpu:
> Mar 13 06:30:35 node1 kernel: [902670.961435] CPU 0: hi: 0, btch: 1 usd: 0
> Mar 13 06:30:35 node1 kernel: [902670.961436] CPU 1: hi: 0, btch: 1 usd: 0
> Mar 13 06:30:35 node1 kernel: [902670.961438] Normal per-cpu:
> Mar 13 06:30:35 node1 kernel: [902670.961439] CPU 0: hi: 186, btch: 31 usd: 180
> Mar 13 06:30:35 node1 kernel: [902670.961441] CPU 1: hi: 186, btch: 31 usd: 136
> Mar 13 06:30:35 node1 kernel: [902670.961442] HighMem per-cpu:
> Mar 13 06:30:35 node1 kernel: [902670.961443] CPU 0: hi: 186, btch: 31 usd: 47
> Mar 13 06:30:35 node1 kernel: [902670.961445] CPU 1: hi: 186, btch: 31 usd: 2
> Mar 13 06:30:35 node1 kernel: [902670.961448] active_anon:137306 inactive_anon:39520 isolated_anon:0
> Mar 13 06:30:35 node1 kernel: [902670.961449] active_file:223625 inactive_file:375487 isolated_file:32
> Mar 13 06:30:35 node1 kernel: [902670.961450] unevictable:4083 dirty:1745 writeback:0 unstable:0
> Mar 13 06:30:35 node1 kernel: [902670.961451] free:187543 slab_reclaimable:50120 slab_unreclaimable:3993
> Mar 13 06:30:35 node1 kernel: [902670.961451] mapped:23637 shmem:11538 pagetables:2294 bounce:52
> Mar 13 06:30:35 node1 kernel: [902670.961456] DMA free:3524kB min:64kB low:80kB high:96kB active_anon:0kB inactive_anon:0kB active_file:2728kB
> inactive_file:6180kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15868kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB
> slab_reclaimable:3512kB slab_unreclaimable:8kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0
> all_unreclaimable? no
> Mar 13 06:30:35 node1 kernel: [902670.961460] lowmem_reserve[]: 0 865 4030 4030
> Mar 13 06:30:35 node1 kernel: [902670.961466] Normal free:40300kB min:3728kB low:4660kB high:5592kB active_anon:4280kB inactive_anon:15772kB
> active_file:199036kB inactive_file:355444kB unevictable:0kB isolated(anon):0kB isolated(file):128kB present:885944kB mlocked:0kB dirty:228kB
> writeback:0kB mapped:24kB shmem:8kB slab_reclaimable:196968kB slab_unreclaimable:15964kB kernel_stack:2608kB pagetables:104kB unstable:0kB
> bounce:208kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
> Mar 13 06:30:35 node1 kernel: [902670.961470] lowmem_reserve[]: 0 0 25321 25321
> Mar 13 06:30:35 node1 kernel: [902670.961476] HighMem free:706348kB min:512kB low:3920kB high:7332kB active_anon:544944kB inactive_anon:142308kB
> active_file:692736kB inactive_file:1140324kB unevictable:16332kB isolated(anon):0kB isolated(file):0kB present:3241132kB mlocked:16332kB dirty:6752kB
> writeback:0kB mapped:94524kB shmem:46144kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:9072kB unstable:0kB bounce:0kB
> writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
> Mar 13 06:30:35 node1 kernel: [902670.961480] lowmem_reserve[]: 0 0 0 0
> Mar 13 06:30:35 node1 kernel: [902670.961483] DMA: 15*4kB 15*8kB 9*16kB 22*32kB 7*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 1*2048kB 0*4096kB = 3524kB
> Mar 13 06:30:35 node1 kernel: [902670.961489] Normal: 3507*4kB 3158*8kB 39*16kB 8*32kB 1*64kB 1*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 40364kB
> Mar 13 06:30:35 node1 kernel: [902670.961495] HighMem: 77*4kB 57*8kB 23*16kB 12*32kB 2057*64kB 2550*128kB 704*256kB 104*512kB 13*1024kB 0*2048kB
> 0*4096kB = 706348kB
> Mar 13 06:30:35 node1 kernel: [902670.961502] 612188 total pagecache pages
> Mar 13 06:30:35 node1 kernel: [902670.961503] 160 pages in swap cache
> Mar 13 06:30:35 node1 kernel: [902670.961505] Swap cache stats: add 21929, delete 21769, find 243894/245286
> Mar 13 06:30:35 node1 kernel: [902670.961506] Free swap = 4880236kB
> Mar 13 06:30:35 node1 kernel: [902670.961507] Total swap = 4883640kB
> Mar 13 06:30:35 node1 kernel: [902670.970841] 1245184 pages RAM
> Mar 13 06:30:35 node1 kernel: [902670.970842] 1017858 pages HighMem
> Mar 13 06:30:35 node1 kernel: [902670.970843] 210949 pages reserved
> Mar 13 06:30:35 node1 kernel: [902670.970844] 327209 pages shared
> Mar 13 06:30:35 node1 kernel: [902670.970845] 642318 pages non-shared
> Mar 13 06:30:35 node1 kernel: [902670.970862] drbd1_receiver: page allocation failure. order:4, mode:0x4020
> Mar 13 06:30:35 node1 kernel: [902670.970865] Pid: 25245, comm: drbd1_receiver Not tainted 2.6.32-5-vserver-686-bigmem #1
> Mar 13 06:30:35 node1 kernel: [902670.970866] Call Trace:
> Mar 13 06:30:35 node1 kernel: [902670.970872] [<c109ad8c>] ? __alloc_pages_nodemask+0x484/0x4d9
> Mar 13 06:30:35 node1 kernel: [902670.970875] [<c109aded>] ? __get_free_pages+0xc/0x17
> Mar 13 06:30:35 node1 kernel: [902670.970877] [<c10bf261>] ? __kmalloc+0x30/0x128
> Mar 13 06:30:35 node1 kernel: [902670.970886] [<c11ee2f4>] ? pskb_expand_head+0x4f/0x157
> Mar 13 06:30:35 node1 kernel: [902670.970889] [<c11ee74b>] ? __pskb_pull_tail+0x40/0x200
> Mar 13 06:30:35 node1 kernel: [902670.970891] [<c11eb077>] ? sock_wfree+0x17/0x4b
> Mar 13 06:30:35 node1 kernel: [902670.970893] [<c11f555e>] ? dev_queue_xmit+0xe4/0x38e
> Mar 13 06:30:35 node1 kernel: [902670.970896] [<c1216cea>] ? ip_finish_output2+0x187/0x1c2
> Mar 13 06:30:35 node1 kernel: [902670.970898] [<c12161ed>] ? ip_local_out+0x15/0x17
> Mar 13 06:30:35 node1 kernel: [902670.970901] [<c12169cc>] ? ip_queue_xmit+0x31d/0x378
> Mar 13 06:30:35 node1 kernel: [902670.970908] [<f81b7696>] ? scsi_init_sgtable+0x2d/0x46 [scsi_mod]
> Mar 13 06:30:35 node1 kernel: [902670.970911] [<c1225c52>] ? tcp_transmit_skb+0x595/0x5cc
> Mar 13 06:30:35 node1 kernel: [902670.970913] [<c1227cb5>] ? tcp_write_xmit+0x7a3/0x874
> Mar 13 06:30:35 node1 kernel: [902670.970915] [<c12236e4>] ? tcp_ack+0x1611/0x1802
> Mar 13 06:30:35 node1 kernel: [902670.970918] [<c122448c>] ? tcp_rcv_established+0x5cf/0x62c
> Mar 13 06:30:35 node1 kernel: [902670.970920] [<c1224fdf>] ? tcp_established_options+0x1d/0x8b
> Mar 13 06:30:35 node1 kernel: [902670.970922] [<c12252a3>] ? tcp_current_mss+0x38/0x53
> Mar 13 06:30:35 node1 kernel: [902670.970924] [<c1227dcc>] ? __tcp_push_pending_frames+0x1e/0x50
> Mar 13 06:30:35 node1 kernel: [902670.970926] [<c12238f0>] ? tcp_data_snd_check+0x1b/0xd2
> Mar 13 06:30:35 node1 kernel: [902670.970928] [<c1224493>] ? tcp_rcv_established+0x5d6/0x62c
> Mar 13 06:30:35 node1 kernel: [902670.970931] [<c122a9ae>] ? tcp_v4_do_rcv+0x15f/0x2cf
> Mar 13 06:30:35 node1 kernel: [902670.970933] [<c11f2e36>] ? net_tx_action+0xa9/0xf9
> Mar 13 06:30:35 node1 kernel: [902670.970936] [<c11ea5ff>] ? release_sock+0x38/0x89
> Mar 13 06:30:35 node1 kernel: [902670.970938] [<c11eab8a>] ? sk_wait_data+0x55/0xa2
> Mar 13 06:30:35 node1 kernel: [902670.970941] [<c104b08a>] ? autoremove_wake_function+0x0/0x2d
> Mar 13 06:30:35 node1 kernel: [902670.970943] [<c121cf09>] ? tcp_recvmsg+0x3e4/0x8e5
> Mar 13 06:30:35 node1 kernel: [902670.970946] [<c11e9cdf>] ? sock_common_recvmsg+0x2f/0x45
> Mar 13 06:30:35 node1 kernel: [902670.970948] [<c11e9cdf>] ? sock_common_recvmsg+0x2f/0x45
> Mar 13 06:30:35 node1 kernel: [902670.970952] [<c11e7fc3>] ? __sock_recvmsg+0x50/0xbb
> Mar 13 06:30:35 node1 kernel: [902670.970954] [<c11e855d>] ? sock_recvmsg+0x9d/0xb6
> Mar 13 06:30:35 node1 kernel: [902670.970956] [<c104b08a>] ? autoremove_wake_function+0x0/0x2d
> Mar 13 06:30:35 node1 kernel: [902670.970959] [<c113c658>] ? __make_request+0x364/0x3d9
> Mar 13 06:30:35 node1 kernel: [902670.970965] [<f9067ddf>] ? drbd_recv+0x62/0x106 [drbd]
> Mar 13 06:30:35 node1 kernel: [902670.970969] [<f9067e95>] ? drbd_recv_header+0x12/0xa5 [drbd]
> Mar 13 06:30:35 node1 kernel: [902670.970974] [<f906e669>] ? drbdd_init+0xf3/0x245 [drbd]
> Mar 13 06:30:35 node1 kernel: [902670.970979] [<f9079bf7>] ? drbd_thread_setup+0x1f/0xcc [drbd]
> Mar 13 06:30:35 node1 kernel: [902670.970983] [<f9079bd8>] ? drbd_thread_setup+0x0/0xcc [drbd]
> Mar 13 06:30:35 node1 kernel: [902670.970985] [<c104ae58>] ? kthread+0x61/0x66
> Mar 13 06:30:35 node1 kernel: [902670.970987] [<c104adf7>] ? kthread+0x0/0x66
> Mar 13 06:30:35 node1 kernel: [902670.970990] [<c1008d87>] ? kernel_thread_helper+0x7/0x10
> Mar 13 06:30:35 node1 kernel: [902670.970992] Mem-Info:
> Mar 13 06:30:35 node1 kernel: [902670.970993] DMA per-cpu:
> Mar 13 06:30:35 node1 kernel: [902670.970994] CPU 0: hi: 0, btch: 1 usd: 0
> Mar 13 06:30:35 node1 kernel: [902670.970996] CPU 1: hi: 0, btch: 1 usd: 0
> Mar 13 06:30:35 node1 kernel: [902670.970997] Normal per-cpu:
> Mar 13 06:30:35 node1 kernel: [902670.970998] CPU 0: hi: 186, btch: 31 usd: 180
> Mar 13 06:30:35 node1 kernel: [902670.971000] CPU 1: hi: 186, btch: 31 usd: 135
> Mar 13 06:30:35 node1 kernel: [902670.971001] HighMem per-cpu:
> Mar 13 06:30:35 node1 kernel: [902670.971002] CPU 0: hi: 186, btch: 31 usd: 47
> Mar 13 06:30:35 node1 kernel: [902670.971004] CPU 1: hi: 186, btch: 31 usd: 56
> Mar 13 06:30:35 node1 kernel: [902670.971007] active_anon:137306 inactive_anon:39520 isolated_anon:0
> Mar 13 06:30:35 node1 kernel: [902670.971008] active_file:223625 inactive_file:375487 isolated_file:32
> Mar 13 06:30:35 node1 kernel: [902670.971009] unevictable:4083 dirty:1745 writeback:0 unstable:0
> Mar 13 06:30:35 node1 kernel: [902670.971010] free:187481 slab_reclaimable:50120 slab_unreclaimable:3993
> Mar 13 06:30:35 node1 kernel: [902670.971011] mapped:23637 shmem:11538 pagetables:2294 bounce:52
> Mar 13 06:30:35 node1 kernel: [902670.971015] DMA free:3524kB min:64kB low:80kB high:96kB active_anon:0kB inactive_anon:0kB active_file:2728kB
> inactive_file:6180kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15868kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB
> slab_reclaimable:3512kB slab_unreclaimable:8kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0
> all_unreclaimable? no
> Mar 13 06:30:35 node1 kernel: [902670.971019] lowmem_reserve[]: 0 865 4030 4030
> Mar 13 06:30:35 node1 kernel: [902670.971025] Normal free:40300kB min:3728kB low:4660kB high:5592kB active_anon:4280kB inactive_anon:15772kB
> active_file:199036kB inactive_file:355444kB unevictable:0kB isolated(anon):0kB isolated(file):128kB present:885944kB mlocked:0kB dirty:228kB
> writeback:0kB mapped:24kB shmem:8kB slab_reclaimable:196968kB slab_unreclaimable:15964kB kernel_stack:2608kB pagetables:104kB unstable:0kB
> bounce:208kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
> Mar 13 06:30:35 node1 kernel: [902670.971029] lowmem_reserve[]: 0 0 25321 25321
> Mar 13 06:30:35 node1 kernel: [902670.971035] HighMem free:706100kB min:512kB low:3920kB high:7332kB active_anon:544944kB inactive_anon:142308kB
> active_file:692736kB inactive_file:1140324kB unevictable:16332kB isolated(anon):0kB isolated(file):0kB present:3241132kB mlocked:16332kB dirty:6752kB
> writeback:0kB mapped:94524kB shmem:46144kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:9072kB unstable:0kB bounce:0kB
> writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
> Mar 13 06:30:35 node1 kernel: [902670.971039] lowmem_reserve[]: 0 0 0 0
> Mar 13 06:30:35 node1 kernel: [902670.971042] DMA: 15*4kB 15*8kB 9*16kB 22*32kB 7*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 1*2048kB 0*4096kB = 3524kB
> Mar 13 06:30:35 node1 kernel: [902670.971048] Normal: 3507*4kB 3158*8kB 39*16kB 8*32kB 1*64kB 1*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 40364kB
> Mar 13 06:30:35 node1 kernel: [902670.971054] HighMem: 47*4kB 57*8kB 23*16kB 12*32kB 2055*64kB 2550*128kB 704*256kB 104*512kB 13*1024kB 0*2048kB
> 0*4096kB = 706100kB
> Mar 13 06:30:35 node1 kernel: [902670.971061] 612188 total pagecache pages
> Mar 13 06:30:35 node1 kernel: [902670.971062] 160 pages in swap cache
> Mar 13 06:30:35 node1 kernel: [902670.971064] Swap cache stats: add 21929, delete 21769, find 243894/245286
> Mar 13 06:30:35 node1 kernel: [902670.971065] Free swap = 4880236kB
> Mar 13 06:30:35 node1 kernel: [902670.971066] Total swap = 4883640kB
> Mar 13 06:30:36 node1 kernel: [902670.980377] 1245184 pages RAM
> Mar 13 06:30:36 node1 kernel: [902670.980378] 1017858 pages HighMem
> Mar 13 06:30:36 node1 kernel: [902670.980379] 210949 pages reserved
> Mar 13 06:30:36 node1 kernel: [902670.980380] 327217 pages shared
> Mar 13 06:30:36 node1 kernel: [902670.980381] 642319 pages non-shared
> Mar 13 06:30:36 node1 kernel: [902670.980397] drbd1_receiver: page allocation failure. order:4, mode:0x4020
> Mar 13 06:30:36 node1 kernel: [902670.980399] Pid: 25245, comm: drbd1_receiver Not tainted 2.6.32-5-vserver-686-bigmem #1
> Mar 13 06:30:36 node1 kernel: [902670.980401] Call Trace:
> Mar 13 06:30:36 node1 kernel: [902670.980406] [<c109ad8c>] ? __alloc_pages_nodemask+0x484/0x4d9
> Mar 13 06:30:36 node1 kernel: [902670.980409] [<c109aded>] ? __get_free_pages+0xc/0x17
> Mar 13 06:30:36 node1 kernel: [902670.980412] [<c10bf261>] ? __kmalloc+0x30/0x128
> Mar 13 06:30:36 node1 kernel: [902670.980422] [<c11ee2f4>] ? pskb_expand_head+0x4f/0x157
> Mar 13 06:30:36 node1 kernel: [902670.980424] [<c11ee74b>] ? __pskb_pull_tail+0x40/0x200
> Mar 13 06:30:36 node1 kernel: [902670.980427] [<c11eb077>] ? sock_wfree+0x17/0x4b
> Mar 13 06:30:36 node1 kernel: [902670.980429] [<c11f555e>] ? dev_queue_xmit+0xe4/0x38e
> Mar 13 06:30:36 node1 kernel: [902670.980432] [<c1216cea>] ? ip_finish_output2+0x187/0x1c2
> Mar 13 06:30:36 node1 kernel: [902670.980434] [<c12161ed>] ? ip_local_out+0x15/0x17
> Mar 13 06:30:36 node1 kernel: [902670.980436] [<c12169cc>] ? ip_queue_xmit+0x31d/0x378
> Mar 13 06:30:36 node1 kernel: [902670.980443] [<f81b7696>] ? scsi_init_sgtable+0x2d/0x46 [scsi_mod]
> Mar 13 06:30:36 node1 kernel: [902670.980447] [<c1225c52>] ? tcp_transmit_skb+0x595/0x5cc
> Mar 13 06:30:36 node1 kernel: [902670.980449] [<c1227cb5>] ? tcp_write_xmit+0x7a3/0x874
> Mar 13 06:30:36 node1 kernel: [902670.980451] [<c12236e4>] ? tcp_ack+0x1611/0x1802
> Mar 13 06:30:36 node1 kernel: [902670.980453] [<c1224fdf>] ? tcp_established_options+0x1d/0x8b
> Mar 13 06:30:36 node1 kernel: [902670.980456] [<c12252a3>] ? tcp_current_mss+0x38/0x53
> Mar 13 06:30:36 node1 kernel: [902670.980458] [<c1227dcc>] ? __tcp_push_pending_frames+0x1e/0x50
> Mar 13 06:30:36 node1 kernel: [902670.980460] [<c12238f0>] ? tcp_data_snd_check+0x1b/0xd2
> Mar 13 06:30:36 node1 kernel: [902670.980462] [<c1224493>] ? tcp_rcv_established+0x5d6/0x62c
> Mar 13 06:30:36 node1 kernel: [902670.980465] [<c122a9ae>] ? tcp_v4_do_rcv+0x15f/0x2cf
> Mar 13 06:30:36 node1 kernel: [902670.980467] [<c11f2e36>] ? net_tx_action+0xa9/0xf9
> Mar 13 06:30:36 node1 kernel: [902670.980469] [<c11ea5ff>] ? release_sock+0x38/0x89
> Mar 13 06:30:36 node1 kernel: [902670.980471] [<c11eab8a>] ? sk_wait_data+0x55/0xa2
> Mar 13 06:30:36 node1 kernel: [902670.980474] [<c104b08a>] ? autoremove_wake_function+0x0/0x2d
> Mar 13 06:30:36 node1 kernel: [902670.980477] [<c121cf09>] ? tcp_recvmsg+0x3e4/0x8e5
> Mar 13 06:30:36 node1 kernel: [902670.980479] [<c11e9cdf>] ? sock_common_recvmsg+0x2f/0x45
> Mar 13 06:30:36 node1 kernel: [902670.980481] [<c11e9cdf>] ? sock_common_recvmsg+0x2f/0x45
> Mar 13 06:30:36 node1 kernel: [902670.980486] [<c11e7fc3>] ? __sock_recvmsg+0x50/0xbb
> Mar 13 06:30:36 node1 kernel: [902670.980488] [<c11e855d>] ? sock_recvmsg+0x9d/0xb6
> Mar 13 06:30:36 node1 kernel: [902670.980490] [<c104b08a>] ? autoremove_wake_function+0x0/0x2d
> Mar 13 06:30:36 node1 kernel: [902670.980494] [<c113c658>] ? __make_request+0x364/0x3d9
> Mar 13 06:30:36 node1 kernel: [902670.980499] [<f9067ddf>] ? drbd_recv+0x62/0x106 [drbd]
> Mar 13 06:30:36 node1 kernel: [902670.980504] [<f9067e95>] ? drbd_recv_header+0x12/0xa5 [drbd]
> Mar 13 06:30:36 node1 kernel: [902670.980508] [<f906e669>] ? drbdd_init+0xf3/0x245 [drbd]
> Mar 13 06:30:36 node1 kernel: [902670.980513] [<f9079bf7>] ? drbd_thread_setup+0x1f/0xcc [drbd]
> Mar 13 06:30:36 node1 kernel: [902670.980517] [<f9079bd8>] ? drbd_thread_setup+0x0/0xcc [drbd]
> Mar 13 06:30:36 node1 kernel: [902670.980519] [<c104ae58>] ? kthread+0x61/0x66
> Mar 13 06:30:36 node1 kernel: [902670.980522] [<c104adf7>] ? kthread+0x0/0x66
> Mar 13 06:30:36 node1 kernel: [902670.980525] [<c1008d87>] ? kernel_thread_helper+0x7/0x10
> Mar 13 06:30:36 node1 kernel: [902670.980526] Mem-Info:
> Mar 13 06:30:36 node1 kernel: [902670.980527] DMA per-cpu:
> Mar 13 06:30:36 node1 kernel: [902670.980529] CPU 0: hi: 0, btch: 1 usd: 0
> Mar 13 06:30:36 node1 kernel: [902670.980530] CPU 1: hi: 0, btch: 1 usd: 0
> Mar 13 06:30:36 node1 kernel: [902670.980531] Normal per-cpu:
> Mar 13 06:30:36 node1 kernel: [902670.980533] CPU 0: hi: 186, btch: 31 usd: 180
> Mar 13 06:30:36 node1 kernel: [902670.980534] CPU 1: hi: 186, btch: 31 usd: 135
> Mar 13 06:30:36 node1 kernel: [902670.980535] HighMem per-cpu:
> Mar 13 06:30:36 node1 kernel: [902670.980537] CPU 0: hi: 186, btch: 31 usd: 47
> Mar 13 06:30:36 node1 kernel: [902670.980538] CPU 1: hi: 186, btch: 31 usd: 50
> Mar 13 06:30:36 node1 kernel: [902670.980542] active_anon:137306 inactive_anon:39520 isolated_anon:0
> Mar 13 06:30:36 node1 kernel: [902670.980542] active_file:223625 inactive_file:375487 isolated_file:32
> Mar 13 06:30:36 node1 kernel: [902670.980543] unevictable:4083 dirty:1745 writeback:0 unstable:0
> Mar 13 06:30:36 node1 kernel: [902670.980544] free:187481 slab_reclaimable:50120 slab_unreclaimable:3993
> Mar 13 06:30:36 node1 kernel: [902670.980545] mapped:23637 shmem:11538 pagetables:2294 bounce:52
> Mar 13 06:30:36 node1 kernel: [902670.980550] DMA free:3524kB min:64kB low:80kB high:96kB active_anon:0kB inactive_anon:0kB active_file:2728kB
> inactive_file:6180kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15868kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB
> slab_reclaimable:3512kB slab_unreclaimable:8kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0
> all_unreclaimable? no
> Mar 13 06:30:36 node1 kernel: [902670.980554] lowmem_reserve[]: 0 865 4030 4030
> Mar 13 06:30:36 node1 kernel: [902670.980560] Normal free:40300kB min:3728kB low:4660kB high:5592kB active_anon:4280kB inactive_anon:15772kB
> active_file:199036kB inactive_file:355444kB unevictable:0kB isolated(anon):0kB isolated(file):128kB present:885944kB mlocked:0kB dirty:228kB
> writeback:0kB mapped:24kB shmem:8kB slab_reclaimable:196968kB slab_unreclaimable:15964kB kernel_stack:2608kB pagetables:104kB unstable:0kB
> bounce:208kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
> Mar 13 06:30:36 node1 kernel: [902670.980564] lowmem_reserve[]: 0 0 25321 25321
> Mar 13 06:30:36 node1 kernel: [902670.980569] HighMem free:706100kB min:512kB low:3920kB high:7332kB active_anon:544944kB inactive_anon:142308kB
> active_file:692736kB inactive_file:1140324kB unevictable:16332kB isolated(anon):0kB isolated(file):0kB present:3241132kB mlocked:16332kB dirty:6752kB
> writeback:0kB mapped:94524kB shmem:46144kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:9072kB unstable:0kB bounce:0kB
> writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
> Mar 13 06:30:36 node1 kernel: [902670.980574] lowmem_reserve[]: 0 0 0 0
> Mar 13 06:30:36 node1 kernel: [902670.980576] DMA: 15*4kB 15*8kB 9*16kB 22*32kB 7*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 1*2048kB 0*4096kB = 3524kB
> Mar 13 06:30:36 node1 kernel: [902670.980583] Normal: 3507*4kB 3158*8kB 39*16kB 8*32kB 1*64kB 1*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 40364kB
> Mar 13 06:30:36 node1 kernel: [902670.980589] HighMem: 47*4kB 57*8kB 23*16kB 12*32kB 2055*64kB 2550*128kB 704*256kB 104*512kB 13*1024kB 0*2048kB
> 0*4096kB = 706100kB
> Mar 13 06:30:36 node1 kernel: [902670.980595] 612188 total pagecache pages
> Mar 13 06:30:36 node1 kernel: [902670.980597] 160 pages in swap cache
> Mar 13 06:30:36 node1 kernel: [902670.980598] Swap cache stats: add 21929, delete 21769, find 243894/245286
> Mar 13 06:30:36 node1 kernel: [902670.980599] Free swap = 4880236kB
> Mar 13 06:30:36 node1 kernel: [902670.980601] Total swap = 4883640kB
> Mar 13 06:30:36 node1 kernel: [902670.989961] 1245184 pages RAM
> Mar 13 06:30:36 node1 kernel: [902670.989963] 1017858 pages HighMem
> Mar 13 06:30:36 node1 kernel: [902670.989964] 210949 pages reserved
> Mar 13 06:30:36 node1 kernel: [902670.989965] 327213 pages shared
> Mar 13 06:30:36 node1 kernel: [902670.989966] 642319 pages non-shared
> Mar 13 06:30:42 node1 kernel: [902677.037409] __ratelimit: 80 callbacks suppressed
> Mar 13 06:30:42 node1 kernel: [902677.037413] drbd1_receiver: page allocation failure. order:4, mode:0x4020
> Mar 13 06:30:42 node1 kernel: [902677.037417] Pid: 25245, comm: drbd1_receiver Not tainted 2.6.32-5-vserver-686-bigmem #1
> Mar 13 06:30:42 node1 kernel: [902677.037420] Call Trace:
> Mar 13 06:30:42 node1 kernel: [902677.037428] [<c109ad8c>] ? __alloc_pages_nodemask+0x484/0x4d9
> Mar 13 06:30:42 node1 kernel: [902677.037433] [<c10255dc>] ? kunmap_atomic+0x48/0x57
> Mar 13 06:30:42 node1 kernel: [902677.037436] [<c109aded>] ? __get_free_pages+0xc/0x17
> Mar 13 06:30:42 node1 kernel: [902677.037440] [<c10bf261>] ? __kmalloc+0x30/0x128
> Mar 13 06:30:42 node1 kernel: [902677.037444] [<c10255dc>] ? kunmap_atomic+0x48/0x57
> Mar 13 06:30:42 node1 kernel: [902677.037448] [<c11ee2f4>] ? pskb_expand_head+0x4f/0x157
> Mar 13 06:30:42 node1 kernel: [902677.037452] [<c11ee74b>] ? __pskb_pull_tail+0x40/0x200
> Mar 13 06:30:42 node1 kernel: [902677.037455] [<c1204d46>] ? __qdisc_run+0x90/0xaa
> Mar 13 06:30:42 node1 kernel: [902677.037459] [<c11f555e>] ? dev_queue_xmit+0xe4/0x38e
> Mar 13 06:30:42 node1 kernel: [902677.037463] [<c1216cea>] ? ip_finish_output2+0x187/0x1c2
> Mar 13 06:30:42 node1 kernel: [902677.037466] [<c12161ed>] ? ip_local_out+0x15/0x17
> Mar 13 06:30:42 node1 kernel: [902677.037469] [<c12169cc>] ? ip_queue_xmit+0x31d/0x378
> Mar 13 06:30:42 node1 kernel: [902677.037473] [<c102ec17>] ? find_busiest_group+0x2e9/0x6e6
> Mar 13 06:30:42 node1 kernel: [902677.037477] [<c1099783>] ? free_one_page+0x52/0x5e
> Mar 13 06:30:42 node1 kernel: [902677.037481] [<c1225c52>] ? tcp_transmit_skb+0x595/0x5cc
> Mar 13 06:30:42 node1 kernel: [902677.037485] [<c1227cb5>] ? tcp_write_xmit+0x7a3/0x874
> Mar 13 06:30:42 node1 kernel: [902677.037488] [<c12236e4>] ? tcp_ack+0x1611/0x1802
> Mar 13 06:30:42 node1 kernel: [902677.037492] [<c1224fdf>] ? tcp_established_options+0x1d/0x8b
> Mar 13 06:30:42 node1 kernel: [902677.037495] [<c12252a3>] ? tcp_current_mss+0x38/0x53
> Mar 13 06:30:42 node1 kernel: [902677.037498] [<c1227dcc>] ? __tcp_push_pending_frames+0x1e/0x50
> Mar 13 06:30:42 node1 kernel: [902677.037502] [<c12238f0>] ? tcp_data_snd_check+0x1b/0xd2
> Mar 13 06:30:42 node1 kernel: [902677.037505] [<c1223f8f>] ? tcp_rcv_established+0xd2/0x62c
> Mar 13 06:30:42 node1 kernel: [902677.037509] [<c128afe9>] ? schedule_timeout+0x20/0xb0
> Mar 13 06:30:42 node1 kernel: [902677.037513] [<c122a9ae>] ? tcp_v4_do_rcv+0x15f/0x2cf
> Mar 13 06:30:42 node1 kernel: [902677.037516] [<c11eabce>] ? sk_wait_data+0x99/0xa2
> Mar 13 06:30:42 node1 kernel: [902677.037520] [<c121bf64>] ? tcp_prequeue_process+0x57/0x6b
> Mar 13 06:30:42 node1 kernel: [902677.037524] [<c121cf66>] ? tcp_recvmsg+0x441/0x8e5
> Mar 13 06:30:42 node1 kernel: [902677.037527] [<c11e9cdf>] ? sock_common_recvmsg+0x2f/0x45
> Mar 13 06:30:42 node1 kernel: [902677.037530] [<c11e9cdf>] ? sock_common_recvmsg+0x2f/0x45
> Mar 13 06:30:42 node1 kernel: [902677.037535] [<c11e7fc3>] ? __sock_recvmsg+0x50/0xbb
> Mar 13 06:30:42 node1 kernel: [902677.037538] [<c11e855d>] ? sock_recvmsg+0x9d/0xb6
> Mar 13 06:30:42 node1 kernel: [902677.037543] [<c104b08a>] ? autoremove_wake_function+0x0/0x2d
> Mar 13 06:30:42 node1 kernel: [902677.037547] [<c113c658>] ? __make_request+0x364/0x3d9
> Mar 13 06:30:42 node1 kernel: [902677.037556] [<f9067ddf>] ? drbd_recv+0x62/0x106 [drbd]
> Mar 13 06:30:42 node1 kernel: [902677.037563] [<f9067e95>] ? drbd_recv_header+0x12/0xa5 [drbd]
> Mar 13 06:30:42 node1 kernel: [902677.037570] [<f906e669>] ? drbdd_init+0xf3/0x245 [drbd]
> Mar 13 06:30:42 node1 kernel: [902677.037577] [<f9079bf7>] ? drbd_thread_setup+0x1f/0xcc [drbd]
> Mar 13 06:30:42 node1 kernel: [902677.037584] [<f9079bd8>] ? drbd_thread_setup+0x0/0xcc [drbd]
> Mar 13 06:30:42 node1 kernel: [902677.037587] [<c104ae58>] ? kthread+0x61/0x66
> Mar 13 06:30:42 node1 kernel: [902677.037590] [<c104adf7>] ? kthread+0x0/0x66
> Mar 13 06:30:42 node1 kernel: [902677.037594] [<c1008d87>] ? kernel_thread_helper+0x7/0x10
> Mar 13 06:30:42 node1 kernel: [902677.037596] Mem-Info:
> Mar 13 06:30:42 node1 kernel: [902677.037598] DMA per-cpu:
> Mar 13 06:30:42 node1 kernel: [902677.037600] CPU 0: hi: 0, btch: 1 usd: 0
> Mar 13 06:30:42 node1 kernel: [902677.037602] CPU 1: hi: 0, btch: 1 usd: 0
> Mar 13 06:30:42 node1 kernel: [902677.037604] Normal per-cpu:
> Mar 13 06:30:42 node1 kernel: [902677.037606] CPU 0: hi: 186, btch: 31 usd: 6
> Mar 13 06:30:42 node1 kernel: [902677.037609] CPU 1: hi: 186, btch: 31 usd: 30
> Mar 13 06:30:42 node1 kernel: [902677.037610] HighMem per-cpu:
> Mar 13 06:30:42 node1 kernel: [902677.037612] CPU 0: hi: 186, btch: 31 usd: 0
> Mar 13 06:30:42 node1 kernel: [902677.037615] CPU 1: hi: 186, btch: 31 usd: 0
> Mar 13 06:30:42 node1 kernel: [902677.037619] active_anon:137310 inactive_anon:39519 isolated_anon:2
> Mar 13 06:30:42 node1 kernel: [902677.037621] active_file:223620 inactive_file:372790 isolated_file:36
> Mar 13 06:30:42 node1 kernel: [902677.037622] unevictable:4083 dirty:3716 writeback:1 unstable:0
> Mar 13 06:30:42 node1 kernel: [902677.037623] free:190562 slab_reclaimable:49977 slab_unreclaimable:4122
> Mar 13 06:30:42 node1 kernel: [902677.037624] mapped:23637 shmem:11538 pagetables:2294 bounce:158
> Mar 13 06:30:42 node1 kernel: [902677.037632] DMA free:3524kB min:64kB low:80kB high:96kB active_anon:0kB inactive_anon:0kB active_file:2728kB
> inactive_file:6180kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15868kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB
> slab_reclaimable:3512kB slab_unreclaimable:8kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0
> all_unreclaimable? no
> Mar 13 06:30:42 node1 kernel: [902677.037637] lowmem_reserve[]: 0 865 4030 4030
> Mar 13 06:30:42 node1 kernel: [902677.037650] Normal free:41960kB min:3728kB low:4660kB high:5592kB active_anon:4276kB inactive_anon:15768kB
> active_file:199060kB inactive_file:354760kB unevictable:0kB isolated(anon):8kB isolated(file):144kB present:885944kB mlocked:0kB dirty:732kB
> writeback:4kB mapped:24kB shmem:8kB slab_reclaimable:196396kB slab_unreclaimable:16384kB kernel_stack:2616kB pagetables:104kB unstable:0kB
> bounce:632kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
> Mar 13 06:30:42 node1 kernel: [902677.037654] lowmem_reserve[]: 0 0 25321 25321
> Mar 13 06:30:42 node1 kernel: [902677.037659] HighMem free:716764kB min:512kB low:3920kB high:7332kB active_anon:544964kB inactive_anon:142308kB
> active_file:692692kB inactive_file:1130220kB unevictable:16332kB isolated(anon):0kB isolated(file):0kB present:3241132kB mlocked:16332kB dirty:14132kB
> writeback:0kB mapped:94524kB shmem:46144kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:9072kB unstable:0kB bounce:0kB
> writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
> Mar 13 06:30:42 node1 kernel: [902677.037664] lowmem_reserve[]: 0 0 0 0
> Mar 13 06:30:42 node1 kernel: [902677.037666] DMA: 15*4kB 15*8kB 9*16kB 22*32kB 7*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 1*2048kB 0*4096kB = 3524kB
> Mar 13 06:30:42 node1 kernel: [902677.037673] Normal: 3610*4kB 3223*8kB 69*16kB 16*32kB 1*64kB 1*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB =
> 42032kB
> Mar 13 06:30:42 node1 kernel: [902677.037679] HighMem: 107*4kB 560*8kB 567*16kB 361*32kB 1846*64kB 2550*128kB 704*256kB 104*512kB 13*1024kB 0*2048kB
> 0*4096kB = 716860kB
> Mar 13 06:30:42 node1 kernel: [902677.037686] 609506 total pagecache pages
> Mar 13 06:30:42 node1 kernel: [902677.037687] 169 pages in swap cache
> Mar 13 06:30:42 node1 kernel: [902677.037688] Swap cache stats: add 21938, delete 21769, find 243894/245286
> Mar 13 06:30:42 node1 kernel: [902677.037690] Free swap = 4880200kB
> Mar 13 06:30:42 node1 kernel: [902677.037691] Total swap = 4883640kB
> Mar 13 06:30:42 node1 kernel: [902677.047173] 1245184 pages RAM
> Mar 13 06:30:42 node1 kernel: [902677.047174] 1017858 pages HighMem
> Mar 13 06:30:42 node1 kernel: [902677.047175] 210949 pages reserved
> Mar 13 06:30:42 node1 kernel: [902677.047176] 328890 pages shared
> Mar 13 06:30:42 node1 kernel: [902677.047177] 637938 pages non-shared
> Mar 13 06:30:42 node1 kernel: [902677.047193] drbd1_receiver: page allocation failure. order:4, mode:0x4020
> Mar 13 06:30:42 node1 kernel: [902677.047196] Pid: 25245, comm: drbd1_receiver Not tainted 2.6.32-5-vserver-686-bigmem #1
> Mar 13 06:30:42 node1 kernel: [902677.047198] Call Trace:
> Mar 13 06:30:42 node1 kernel: [902677.047204] [<c109ad8c>] ? __alloc_pages_nodemask+0x484/0x4d9
> Mar 13 06:30:42 node1 kernel: [902677.047208] [<c10255dc>] ? kunmap_atomic+0x48/0x57
> Mar 13 06:30:42 node1 kernel: [902677.047210] [<c109aded>] ? __get_free_pages+0xc/0x17
> Mar 13 06:30:42 node1 kernel: [902677.047213] [<c10bf261>] ? __kmalloc+0x30/0x128
> Mar 13 06:30:42 node1 kernel: [902677.047215] [<c10255dc>] ? kunmap_atomic+0x48/0x57
> Mar 13 06:30:42 node1 kernel: [902677.047218] [<c11ee2f4>] ? pskb_expand_head+0x4f/0x157
> Mar 13 06:30:42 node1 kernel: [902677.047221] [<c11ee74b>] ? __pskb_pull_tail+0x40/0x200
> Mar 13 06:30:42 node1 kernel: [902677.047223] [<c11eb077>] ? sock_wfree+0x17/0x4b
> Mar 13 06:30:42 node1 kernel: [902677.047226] [<c11f555e>] ? dev_queue_xmit+0xe4/0x38e
> Mar 13 06:30:42 node1 kernel: [902677.047229] [<c1216cea>] ? ip_finish_output2+0x187/0x1c2
> Mar 13 06:30:42 node1 kernel: [902677.047231] [<c12161ed>] ? ip_local_out+0x15/0x17
> Mar 13 06:30:42 node1 kernel: [902677.047233] [<c12169cc>] ? ip_queue_xmit+0x31d/0x378
> Mar 13 06:30:42 node1 kernel: [902677.047236] [<c102ec17>] ? find_busiest_group+0x2e9/0x6e6
> Mar 13 06:30:42 node1 kernel: [902677.047238] [<c1099783>] ? free_one_page+0x52/0x5e
> Mar 13 06:30:42 node1 kernel: [902677.047241] [<c1225c52>] ? tcp_transmit_skb+0x595/0x5cc
> Mar 13 06:30:42 node1 kernel: [902677.047243] [<c1227cb5>] ? tcp_write_xmit+0x7a3/0x874
> Mar 13 06:30:42 node1 kernel: [902677.047245] [<c12236e4>] ? tcp_ack+0x1611/0x1802
> Mar 13 06:30:42 node1 kernel: [902677.047248] [<c1224fdf>] ? tcp_established_options+0x1d/0x8b
> Mar 13 06:30:42 node1 kernel: [902677.047250] [<c12252a3>] ? tcp_current_mss+0x38/0x53
> Mar 13 06:30:42 node1 kernel: [902677.047252] [<c1227dcc>] ? __tcp_push_pending_frames+0x1e/0x50
> Mar 13 06:30:42 node1 kernel: [902677.047254] [<c12238f0>] ? tcp_data_snd_check+0x1b/0xd2
> Mar 13 06:30:42 node1 kernel: [902677.047256] [<c1224493>] ? tcp_rcv_established+0x5d6/0x62c
> Mar 13 06:30:42 node1 kernel: [902677.047260] [<c128afe9>] ? schedule_timeout+0x20/0xb0
> Mar 13 06:30:42 node1 kernel: [902677.047262] [<c122a9ae>] ? tcp_v4_do_rcv+0x15f/0x2cf
> Mar 13 06:30:42 node1 kernel: [902677.047264] [<c11eabce>] ? sk_wait_data+0x99/0xa2
> Mar 13 06:30:42 node1 kernel: [902677.047267] [<c121bf64>] ? tcp_prequeue_process+0x57/0x6b
> Mar 13 06:30:42 node1 kernel: [902677.047269] [<c121cf66>] ? tcp_recvmsg+0x441/0x8e5
> Mar 13 06:30:42 node1 kernel: [902677.047272] [<c11e9cdf>] ? sock_common_recvmsg+0x2f/0x45
> Mar 13 06:30:42 node1 kernel: [902677.047274] [<c11e9cdf>] ? sock_common_recvmsg+0x2f/0x45
> Mar 13 06:30:42 node1 kernel: [902677.047277] [<c11e7fc3>] ? __sock_recvmsg+0x50/0xbb
> Mar 13 06:30:42 node1 kernel: [902677.047279] [<c11e855d>] ? sock_recvmsg+0x9d/0xb6
> Mar 13 06:30:42 node1 kernel: [902677.047282] [<c104b08a>] ? autoremove_wake_function+0x0/0x2d
> Mar 13 06:30:42 node1 kernel: [902677.047286] [<c113c658>] ? __make_request+0x364/0x3d9
> Mar 13 06:30:42 node1 kernel: [902677.047291] [<f9067ddf>] ? drbd_recv+0x62/0x106 [drbd]
> Mar 13 06:30:42 node1 kernel: [902677.047296] [<f9067e95>] ? drbd_recv_header+0x12/0xa5 [drbd]
> Mar 13 06:30:42 node1 kernel: [902677.047301] [<f906e669>] ? drbdd_init+0xf3/0x245 [drbd]
> Mar 13 06:30:42 node1 kernel: [902677.047305] [<f9079bf7>] ? drbd_thread_setup+0x1f/0xcc [drbd]
> Mar 13 06:30:42 node1 kernel: [902677.047310] [<f9079bd8>] ? drbd_thread_setup+0x0/0xcc [drbd]
> Mar 13 06:30:42 node1 kernel: [902677.047312] [<c104ae58>] ? kthread+0x61/0x66
> Mar 13 06:30:42 node1 kernel: [902677.047314] [<c104adf7>] ? kthread+0x0/0x66
> Mar 13 06:30:42 node1 kernel: [902677.047317] [<c1008d87>] ? kernel_thread_helper+0x7/0x10
> Mar 13 06:30:42 node1 kernel: [902677.047319] Mem-Info:
> Mar 13 06:30:42 node1 kernel: [902677.047320] DMA per-cpu:
> Mar 13 06:30:42 node1 kernel: [902677.047321] CPU 0: hi: 0, btch: 1 usd: 0
> Mar 13 06:30:42 node1 kernel: [902677.047323] CPU 1: hi: 0, btch: 1 usd: 0
> Mar 13 06:30:42 node1 kernel: [902677.047324] Normal per-cpu:
> Mar 13 06:30:42 node1 kernel: [902677.047326] CPU 0: hi: 186, btch: 31 usd: 5
> Mar 13 06:30:42 node1 kernel: [902677.047327] CPU 1: hi: 186, btch: 31 usd: 30
> Mar 13 06:30:42 node1 kernel: [902677.047328] HighMem per-cpu:
> Mar 13 06:30:42 node1 kernel: [902677.047330] CPU 0: hi: 186, btch: 31 usd: 25
> Mar 13 06:30:42 node1 kernel: [902677.047331] CPU 1: hi: 186, btch: 31 usd: 0
> Mar 13 06:30:42 node1 kernel: [902677.047334] active_anon:137310 inactive_anon:39519 isolated_anon:2
> Mar 13 06:30:42 node1 kernel: [902677.047335] active_file:223620 inactive_file:372790 isolated_file:36
> Mar 13 06:30:42 node1 kernel: [902677.047336] unevictable:4083 dirty:3716 writeback:1 unstable:0
> Mar 13 06:30:42 node1 kernel: [902677.047337] free:190531 slab_reclaimable:49977 slab_unreclaimable:4098
> Mar 13 06:30:42 node1 kernel: [902677.047338] mapped:23637 shmem:11538 pagetables:2294 bounce:158
> Mar 13 06:30:42 node1 kernel: [902677.047342] DMA free:3524kB min:64kB low:80kB high:96kB active_anon:0kB inactive_anon:0kB active_file:2728kB
> inactive_file:6180kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15868kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB
> slab_reclaimable:3512kB slab_unreclaimable:8kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0
> all_unreclaimable? no
> Mar 13 06:30:42 node1 kernel: [902677.047346] lowmem_reserve[]: 0 865 4030 4030
> Mar 13 06:30:42 node1 kernel: [902677.047352] Normal free:41960kB min:3728kB low:4660kB high:5592kB active_anon:4276kB inactive_anon:15768kB
> active_file:199060kB inactive_file:354760kB unevictable:0kB isolated(anon):8kB isolated(file):144kB present:885944kB mlocked:0kB dirty:732kB
> writeback:4kB mapped:24kB shmem:8kB slab_reclaimable:196396kB slab_unreclaimable:16384kB kernel_stack:2616kB pagetables:104kB unstable:0kB
> bounce:632kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
> Mar 13 06:30:42 node1 kernel: [902677.047356] lowmem_reserve[]: 0 0 25321 25321
> Mar 13 06:30:42 node1 kernel: [902677.047362] HighMem free:716640kB min:512kB low:3920kB high:7332kB active_anon:544964kB inactive_anon:142308kB
> active_file:692692kB inactive_file:1130220kB unevictable:16332kB isolated(anon):0kB isolated(file):0kB present:3241132kB mlocked:16332kB dirty:14132kB
> writeback:0kB mapped:94524kB shmem:46144kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:9072kB unstable:0kB bounce:0kB
> writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
> Mar 13 06:30:42 node1 kernel: [902677.047366] lowmem_reserve[]: 0 0 0 0
> Mar 13 06:30:42 node1 kernel: [902677.047369] DMA: 15*4kB 15*8kB 9*16kB 22*32kB 7*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 1*2048kB 0*4096kB = 3524kB
> Mar 13 06:30:42 node1 kernel: [902677.047375] Normal: 3610*4kB 3223*8kB 69*16kB 16*32kB 1*64kB 1*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB =
> 42032kB
> Mar 13 06:30:42 node1 kernel: [902677.047382] HighMem: 106*4kB 545*8kB 567*16kB 361*32kB 1846*64kB 2550*128kB 704*256kB 104*512kB 13*1024kB 0*2048kB
> 0*4096kB = 716736kB
> Mar 13 06:30:42 node1 kernel: [902677.047389] 609506 total pagecache pages
> Mar 13 06:30:42 node1 kernel: [902677.047390] 169 pages in swap cache
> Mar 13 06:30:42 node1 kernel: [902677.047391] Swap cache stats: add 21938, delete 21769, find 243894/245286
> Mar 13 06:30:42 node1 kernel: [902677.047393] Free swap = 4880200kB
> Mar 13 06:30:42 node1 kernel: [902677.047394] Total swap = 4883640kB
> Mar 13 06:30:42 node1 kernel: [902677.056949] 1245184 pages RAM
> Mar 13 06:30:42 node1 kernel: [902677.056951] 1017858 pages HighMem
> Mar 13 06:30:42 node1 kernel: [902677.056952] 210949 pages reserved
> Mar 13 06:30:42 node1 kernel: [902677.056953] 328766 pages shared
> Mar 13 06:30:42 node1 kernel: [902677.056954] 637885 pages non-shared
>
> Can someone please direct me where to look?
>