[DRBD-user] Issues after upgrade

Konrad Korzeniowski drbd at pandur.net
Sun Mar 13 16:38:38 CET 2011

Note: "permalinks" may not be as permanent as we would like,
direct links of old sources may well be a few messages off.


Hi,

I hope that some will be able to help me with strange (at least for me) issue after Debian upgrade (Lenny -> Squeeze).

I'm using dual master/slave setup controlled by heartbeat, where the ha resources are whole vservers.
There is dedicated drbd/heartbeat crossover connection between nodes.

My current drbd.conf:
-------------------------------------------------------------------------------------------
global {
    usage-count no;
}
common {
  syncer { rate 10M; }
}
resource ocfs2 {
  protocol C;
  handlers {
    pri-on-incon-degr "echo o > /proc/sysrq-trigger ; halt -f";
    pri-lost-after-sb "echo o > /proc/sysrq-trigger ; halt -f";
    local-io-error "echo o > /proc/sysrq-trigger ; halt -f";
  }
  startup {
  }
  disk {
    on-io-error   detach;
  }
  net {
    allow-two-primaries;
    after-sb-0pri disconnect;
    after-sb-1pri disconnect;
    after-sb-2pri disconnect;
    rr-conflict disconnect;
  }
  syncer {
    rate 10M;
    al-extents 257;
  }
  on node1 {
    device     /dev/drbd0;
    disk       /dev/sda6;
    address    10.10.10.45:7788;
    meta-disk  internal;
  }
  on node2 {
    device    /dev/drbd0;
    disk      /dev/sda6;
    address   10.10.10.46:7788;
    meta-disk internal;
  }
}
resource grupa1-200 {
  protocol C;
  handlers {
    pri-on-incon-degr "echo o > /proc/sysrq-trigger ; halt -f";
    pri-lost-after-sb "echo o > /proc/sysrq-trigger ; halt -f";
    local-io-error "echo o > /proc/sysrq-trigger ; halt -f";
  }
  startup {
  }
  disk {
    on-io-error   detach;
  }
  net {
    after-sb-0pri disconnect;
    after-sb-1pri disconnect;
    after-sb-2pri disconnect;
    rr-conflict disconnect;
         # wartosc 0 oznacza auto
    sndbuf-size 0;
  }
  syncer {
    rate 30M;
    al-extents 257;
  }
  on node1 {
    device     /dev/drbd1;
    disk       /dev/sda7;
    address    10.10.10.45:7789;
    meta-disk  internal;
  }
  on node2 {
    device    /dev/drbd1;
    disk      /dev/sda7;
    address   10.10.10.46:7789;
    meta-disk internal;
  }
}
resource grupa2-200 {
  protocol C;
  handlers {
    pri-on-incon-degr "echo o > /proc/sysrq-trigger ; halt -f";
    pri-lost-after-sb "echo o > /proc/sysrq-trigger ; halt -f";
    local-io-error "echo o > /proc/sysrq-trigger ; halt -f";
  }
  startup {
  }
  disk {
    on-io-error   detach;
  }
  net {
    after-sb-0pri disconnect;
    after-sb-1pri disconnect;
    after-sb-2pri disconnect;
    rr-conflict disconnect;
  }
  syncer {
    rate 30M;
    al-extents 257;
  }
  on node1 {
    device     /dev/drbd2;
    disk       /dev/sdb6;
    address    10.10.10.45:7790;
    meta-disk  internal;
  }
  on node2 {
    device    /dev/drbd2;
    disk      /dev/sdb6;
    address   10.10.10.46:7790;
    meta-disk internal;
  }
}
-------------------------------------------------------------------------------------------

And the problem is that every morning (1 or more times) in logs I can see:

Mar 13 06:30:35 node1 kernel: [902670.893173] __ratelimit: 407 callbacks suppressed
Mar 13 06:30:35 node1 kernel: [902670.893175] drbd1_receiver: page allocation failure. order:4, mode:0x4020
Mar 13 06:30:35 node1 kernel: [902670.893178] Pid: 25245, comm: drbd1_receiver Not tainted 2.6.32-5-vserver-686-bigmem #1
Mar 13 06:30:35 node1 kernel: [902670.893180] Call Trace:
Mar 13 06:30:35 node1 kernel: [902670.893187]  [<c109ad8c>] ? __alloc_pages_nodemask+0x484/0x4d9
Mar 13 06:30:35 node1 kernel: [902670.893190]  [<c109aded>] ? __get_free_pages+0xc/0x17
Mar 13 06:30:35 node1 kernel: [902670.893194]  [<c10bf261>] ? __kmalloc+0x30/0x128
Mar 13 06:30:35 node1 kernel: [902670.893207]  [<c11ee2f4>] ? pskb_expand_head+0x4f/0x157
Mar 13 06:30:35 node1 kernel: [902670.893209]  [<c11ee74b>] ? __pskb_pull_tail+0x40/0x200
Mar 13 06:30:35 node1 kernel: [902670.893212]  [<c11f555e>] ? dev_queue_xmit+0xe4/0x38e
Mar 13 06:30:35 node1 kernel: [902670.893215]  [<c1216cea>] ? ip_finish_output2+0x187/0x1c2
Mar 13 06:30:35 node1 kernel: [902670.893217]  [<c12161ed>] ? ip_local_out+0x15/0x17
Mar 13 06:30:35 node1 kernel: [902670.893219]  [<c12169cc>] ? ip_queue_xmit+0x31d/0x378
Mar 13 06:30:35 node1 kernel: [902670.893222]  [<c1224fdf>] ? tcp_established_options+0x1d/0x8b
Mar 13 06:30:35 node1 kernel: [902670.893224]  [<c1225c52>] ? tcp_transmit_skb+0x595/0x5cc
Mar 13 06:30:35 node1 kernel: [902670.893227]  [<c1227cb5>] ? tcp_write_xmit+0x7a3/0x874
Mar 13 06:30:35 node1 kernel: [902670.893229]  [<c12236e4>] ? tcp_ack+0x1611/0x1802
Mar 13 06:30:35 node1 kernel: [902670.893231]  [<c1224fdf>] ? tcp_established_options+0x1d/0x8b
Mar 13 06:30:35 node1 kernel: [902670.893233]  [<c12252a3>] ? tcp_current_mss+0x38/0x53
Mar 13 06:30:35 node1 kernel: [902670.893236]  [<c1227dcc>] ? __tcp_push_pending_frames+0x1e/0x50
Mar 13 06:30:35 node1 kernel: [902670.893238]  [<c12238f0>] ? tcp_data_snd_check+0x1b/0xd2
Mar 13 06:30:35 node1 kernel: [902670.893240]  [<c1224493>] ? tcp_rcv_established+0x5d6/0x62c
Mar 13 06:30:35 node1 kernel: [902670.893243]  [<c128afe9>] ? schedule_timeout+0x20/0xb0
Mar 13 06:30:35 node1 kernel: [902670.893245]  [<c122a9ae>] ? tcp_v4_do_rcv+0x15f/0x2cf
Mar 13 06:30:35 node1 kernel: [902670.893248]  [<c11eabce>] ? sk_wait_data+0x99/0xa2
Mar 13 06:30:35 node1 kernel: [902670.893250]  [<c121bf64>] ? tcp_prequeue_process+0x57/0x6b
Mar 13 06:30:35 node1 kernel: [902670.893253]  [<c121cf66>] ? tcp_recvmsg+0x441/0x8e5
Mar 13 06:30:35 node1 kernel: [902670.893255]  [<c11e9cdf>] ? sock_common_recvmsg+0x2f/0x45
Mar 13 06:30:35 node1 kernel: [902670.893257]  [<c11e9cdf>] ? sock_common_recvmsg+0x2f/0x45
Mar 13 06:30:35 node1 kernel: [902670.893260]  [<c11e7fc3>] ? __sock_recvmsg+0x50/0xbb
Mar 13 06:30:35 node1 kernel: [902670.893262]  [<c11e855d>] ? sock_recvmsg+0x9d/0xb6
Mar 13 06:30:35 node1 kernel: [902670.893266]  [<c104b08a>] ? autoremove_wake_function+0x0/0x2d
Mar 13 06:30:35 node1 kernel: [902670.893269]  [<c113c658>] ? __make_request+0x364/0x3d9
Mar 13 06:30:35 node1 kernel: [902670.893276]  [<f9067ddf>] ? drbd_recv+0x62/0x106 [drbd]
Mar 13 06:30:35 node1 kernel: [902670.893280]  [<f9067e95>] ? drbd_recv_header+0x12/0xa5 [drbd]
Mar 13 06:30:35 node1 kernel: [902670.893285]  [<f906e669>] ? drbdd_init+0xf3/0x245 [drbd]
Mar 13 06:30:35 node1 kernel: [902670.893290]  [<f9079bf7>] ? drbd_thread_setup+0x1f/0xcc [drbd]
Mar 13 06:30:35 node1 kernel: [902670.893294]  [<f9079bd8>] ? drbd_thread_setup+0x0/0xcc [drbd]
Mar 13 06:30:35 node1 kernel: [902670.893296]  [<c104ae58>] ? kthread+0x61/0x66
Mar 13 06:30:35 node1 kernel: [902670.893299]  [<c104adf7>] ? kthread+0x0/0x66
Mar 13 06:30:35 node1 kernel: [902670.893302]  [<c1008d87>] ? kernel_thread_helper+0x7/0x10
Mar 13 06:30:35 node1 kernel: [902670.893303] Mem-Info:
Mar 13 06:30:35 node1 kernel: [902670.893304] DMA per-cpu:
Mar 13 06:30:35 node1 kernel: [902670.893306] CPU    0: hi:    0, btch:   1 usd:   0
Mar 13 06:30:35 node1 kernel: [902670.893307] CPU    1: hi:    0, btch:   1 usd:   0
Mar 13 06:30:35 node1 kernel: [902670.893309] Normal per-cpu:
Mar 13 06:30:35 node1 kernel: [902670.893310] CPU    0: hi:  186, btch:  31 usd: 173
Mar 13 06:30:35 node1 kernel: [902670.893312] CPU    1: hi:  186, btch:  31 usd: 155
Mar 13 06:30:35 node1 kernel: [902670.893313] HighMem per-cpu:
Mar 13 06:30:35 node1 kernel: [902670.893314] CPU    0: hi:  186, btch:  31 usd:  47
Mar 13 06:30:35 node1 kernel: [902670.893316] CPU    1: hi:  186, btch:  31 usd:  28
Mar 13 06:30:35 node1 kernel: [902670.893319] active_anon:137306 inactive_anon:39520 isolated_anon:0
Mar 13 06:30:35 node1 kernel: [902670.893320]  active_file:223625 inactive_file:375413 isolated_file:32
Mar 13 06:30:35 node1 kernel: [902670.893321]  unevictable:4083 dirty:1708 writeback:0 unstable:0
Mar 13 06:30:35 node1 kernel: [902670.893321]  free:187603 slab_reclaimable:50120 slab_unreclaimable:3975
Mar 13 06:30:35 node1 kernel: [902670.893322]  mapped:23637 shmem:11538 pagetables:2294 bounce:27
Mar 13 06:30:35 node1 kernel: [902670.893327] DMA free:3524kB min:64kB low:80kB high:96kB active_anon:0kB inactive_anon:0kB active_file:2728kB
inactive_file:6180kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15868kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB
slab_reclaimable:3512kB slab_unreclaimable:8kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0
all_unreclaimable? no
Mar 13 06:30:35 node1 kernel: [902670.893331] lowmem_reserve[]: 0 865 4030 4030
Mar 13 06:30:35 node1 kernel: [902670.893337] Normal free:40292kB min:3728kB low:4660kB high:5592kB active_anon:4280kB inactive_anon:15772kB
active_file:199036kB inactive_file:355444kB unevictable:0kB isolated(anon):0kB isolated(file):128kB present:885944kB mlocked:0kB dirty:228kB
writeback:0kB mapped:24kB shmem:8kB slab_reclaimable:196968kB slab_unreclaimable:15892kB kernel_stack:2608kB pagetables:104kB unstable:0kB
bounce:108kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Mar 13 06:30:35 node1 kernel: [902670.893341] lowmem_reserve[]: 0 0 25321 25321
Mar 13 06:30:35 node1 kernel: [902670.893347] HighMem free:706596kB min:512kB low:3920kB high:7332kB active_anon:544944kB inactive_anon:142308kB
active_file:692736kB inactive_file:1140028kB unevictable:16332kB isolated(anon):0kB isolated(file):0kB present:3241132kB mlocked:16332kB dirty:6604kB
writeback:0kB mapped:94524kB shmem:46144kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:9072kB unstable:0kB bounce:0kB
writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Mar 13 06:30:35 node1 kernel: [902670.893351] lowmem_reserve[]: 0 0 0 0
Mar 13 06:30:35 node1 kernel: [902670.893354] DMA: 15*4kB 15*8kB 9*16kB 22*32kB 7*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 1*2048kB 0*4096kB = 3524kB
Mar 13 06:30:35 node1 kernel: [902670.893360] Normal: 3507*4kB 3161*8kB 37*16kB 8*32kB 2*64kB 1*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 40420kB
Mar 13 06:30:35 node1 kernel: [902670.893367] HighMem: 77*4kB 56*8kB 23*16kB 12*32kB 2061*64kB 2550*128kB 704*256kB 104*512kB 13*1024kB 0*2048kB
0*4096kB = 706596kB
Mar 13 06:30:35 node1 kernel: [902670.893373] 612114 total pagecache pages
Mar 13 06:30:35 node1 kernel: [902670.893374] 160 pages in swap cache
Mar 13 06:30:35 node1 kernel: [902670.893376] Swap cache stats: add 21929, delete 21769, find 243894/245286
Mar 13 06:30:35 node1 kernel: [902670.893377] Free swap  = 4880236kB
Mar 13 06:30:35 node1 kernel: [902670.893378] Total swap = 4883640kB
Mar 13 06:30:35 node1 kernel: [902670.902900] 1245184 pages RAM
Mar 13 06:30:35 node1 kernel: [902670.902902] 1017858 pages HighMem
Mar 13 06:30:35 node1 kernel: [902670.902903] 210949 pages reserved
Mar 13 06:30:35 node1 kernel: [902670.902904] 327380 pages shared
Mar 13 06:30:35 node1 kernel: [902670.902905] 642241 pages non-shared
Mar 13 06:30:35 node1 kernel: [902670.902921] drbd1_receiver: page allocation failure. order:4, mode:0x4020
Mar 13 06:30:35 node1 kernel: [902670.902924] Pid: 25245, comm: drbd1_receiver Not tainted 2.6.32-5-vserver-686-bigmem #1
Mar 13 06:30:35 node1 kernel: [902670.902926] Call Trace:
Mar 13 06:30:35 node1 kernel: [902670.902931]  [<c109ad8c>] ? __alloc_pages_nodemask+0x484/0x4d9
Mar 13 06:30:35 node1 kernel: [902670.902934]  [<c109aded>] ? __get_free_pages+0xc/0x17
Mar 13 06:30:35 node1 kernel: [902670.902937]  [<c10bf261>] ? __kmalloc+0x30/0x128
Mar 13 06:30:35 node1 kernel: [902670.902946]  [<c11ee2f4>] ? pskb_expand_head+0x4f/0x157
Mar 13 06:30:35 node1 kernel: [902670.902948]  [<c11ee74b>] ? __pskb_pull_tail+0x40/0x200
Mar 13 06:30:35 node1 kernel: [902670.902950]  [<c11eb077>] ? sock_wfree+0x17/0x4b
Mar 13 06:30:35 node1 kernel: [902670.902953]  [<c11f555e>] ? dev_queue_xmit+0xe4/0x38e
Mar 13 06:30:35 node1 kernel: [902670.902956]  [<c1216cea>] ? ip_finish_output2+0x187/0x1c2
Mar 13 06:30:35 node1 kernel: [902670.902958]  [<c12161ed>] ? ip_local_out+0x15/0x17
Mar 13 06:30:35 node1 kernel: [902670.902961]  [<c12169cc>] ? ip_queue_xmit+0x31d/0x378
Mar 13 06:30:35 node1 kernel: [902670.902963]  [<c1224fdf>] ? tcp_established_options+0x1d/0x8b
Mar 13 06:30:35 node1 kernel: [902670.902966]  [<c1225c52>] ? tcp_transmit_skb+0x595/0x5cc
Mar 13 06:30:35 node1 kernel: [902670.902968]  [<c1227cb5>] ? tcp_write_xmit+0x7a3/0x874
Mar 13 06:30:35 node1 kernel: [902670.902970]  [<c12236e4>] ? tcp_ack+0x1611/0x1802
Mar 13 06:30:35 node1 kernel: [902670.902973]  [<c1224fdf>] ? tcp_established_options+0x1d/0x8b
Mar 13 06:30:35 node1 kernel: [902670.902975]  [<c12252a3>] ? tcp_current_mss+0x38/0x53
Mar 13 06:30:35 node1 kernel: [902670.902977]  [<c1227dcc>] ? __tcp_push_pending_frames+0x1e/0x50
Mar 13 06:30:35 node1 kernel: [902670.902979]  [<c12238f0>] ? tcp_data_snd_check+0x1b/0xd2
Mar 13 06:30:35 node1 kernel: [902670.902982]  [<c1223f8f>] ? tcp_rcv_established+0xd2/0x62c
Mar 13 06:30:35 node1 kernel: [902670.902986]  [<c128afe9>] ? schedule_timeout+0x20/0xb0
Mar 13 06:30:35 node1 kernel: [902670.902989]  [<c122a9ae>] ? tcp_v4_do_rcv+0x15f/0x2cf
Mar 13 06:30:35 node1 kernel: [902670.902991]  [<c11eabce>] ? sk_wait_data+0x99/0xa2
Mar 13 06:30:35 node1 kernel: [902670.902993]  [<c121bf64>] ? tcp_prequeue_process+0x57/0x6b
Mar 13 06:30:35 node1 kernel: [902670.902996]  [<c121cf66>] ? tcp_recvmsg+0x441/0x8e5
Mar 13 06:30:35 node1 kernel: [902670.902998]  [<c11e9cdf>] ? sock_common_recvmsg+0x2f/0x45
Mar 13 06:30:35 node1 kernel: [902670.903000]  [<c11e9cdf>] ? sock_common_recvmsg+0x2f/0x45
Mar 13 06:30:35 node1 kernel: [902670.903003]  [<c11e7fc3>] ? __sock_recvmsg+0x50/0xbb
Mar 13 06:30:35 node1 kernel: [902670.903005]  [<c11e855d>] ? sock_recvmsg+0x9d/0xb6
Mar 13 06:30:35 node1 kernel: [902670.903009]  [<c104b08a>] ? autoremove_wake_function+0x0/0x2d
Mar 13 06:30:35 node1 kernel: [902670.903012]  [<c113c658>] ? __make_request+0x364/0x3d9
Mar 13 06:30:35 node1 kernel: [902670.903019]  [<f9067ddf>] ? drbd_recv+0x62/0x106 [drbd]
Mar 13 06:30:35 node1 kernel: [902670.903023]  [<f9067e95>] ? drbd_recv_header+0x12/0xa5 [drbd]
Mar 13 06:30:35 node1 kernel: [902670.903028]  [<f906e669>] ? drbdd_init+0xf3/0x245 [drbd]
Mar 13 06:30:35 node1 kernel: [902670.903033]  [<f9079bf7>] ? drbd_thread_setup+0x1f/0xcc [drbd]
Mar 13 06:30:35 node1 kernel: [902670.903037]  [<f9079bd8>] ? drbd_thread_setup+0x0/0xcc [drbd]
Mar 13 06:30:35 node1 kernel: [902670.903039]  [<c104ae58>] ? kthread+0x61/0x66
Mar 13 06:30:35 node1 kernel: [902670.903041]  [<c104adf7>] ? kthread+0x0/0x66
Mar 13 06:30:35 node1 kernel: [902670.903044]  [<c1008d87>] ? kernel_thread_helper+0x7/0x10
Mar 13 06:30:35 node1 kernel: [902670.903046] Mem-Info:
Mar 13 06:30:35 node1 kernel: [902670.903048] DMA per-cpu:
Mar 13 06:30:35 node1 kernel: [902670.903049] CPU    0: hi:    0, btch:   1 usd:   0
Mar 13 06:30:35 node1 kernel: [902670.903051] CPU    1: hi:    0, btch:   1 usd:   0
Mar 13 06:30:35 node1 kernel: [902670.903052] Normal per-cpu:
Mar 13 06:30:35 node1 kernel: [902670.903054] CPU    0: hi:  186, btch:  31 usd: 173
Mar 13 06:30:35 node1 kernel: [902670.903055] CPU    1: hi:  186, btch:  31 usd: 154
Mar 13 06:30:35 node1 kernel: [902670.903057] HighMem per-cpu:
Mar 13 06:30:35 node1 kernel: [902670.903058] CPU    0: hi:  186, btch:  31 usd:  47
Mar 13 06:30:35 node1 kernel: [902670.903060] CPU    1: hi:  186, btch:  31 usd:  20
Mar 13 06:30:35 node1 kernel: [902670.903063] active_anon:137306 inactive_anon:39520 isolated_anon:0
Mar 13 06:30:35 node1 kernel: [902670.903064]  active_file:223625 inactive_file:375413 isolated_file:32
Mar 13 06:30:35 node1 kernel: [902670.903065]  unevictable:4083 dirty:1708 writeback:0 unstable:0
Mar 13 06:30:35 node1 kernel: [902670.903066]  free:187603 slab_reclaimable:50120 slab_unreclaimable:3975
Mar 13 06:30:35 node1 kernel: [902670.903066]  mapped:23637 shmem:11538 pagetables:2294 bounce:27
Mar 13 06:30:35 node1 kernel: [902670.903071] DMA free:3524kB min:64kB low:80kB high:96kB active_anon:0kB inactive_anon:0kB active_file:2728kB
inactive_file:6180kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15868kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB
slab_reclaimable:3512kB slab_unreclaimable:8kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0
all_unreclaimable? no
Mar 13 06:30:35 node1 kernel: [902670.903075] lowmem_reserve[]: 0 865 4030 4030
Mar 13 06:30:35 node1 kernel: [902670.903081] Normal free:40292kB min:3728kB low:4660kB high:5592kB active_anon:4280kB inactive_anon:15772kB
active_file:199036kB inactive_file:355444kB unevictable:0kB isolated(anon):0kB isolated(file):128kB present:885944kB mlocked:0kB dirty:228kB
writeback:0kB mapped:24kB shmem:8kB slab_reclaimable:196968kB slab_unreclaimable:15892kB kernel_stack:2608kB pagetables:104kB unstable:0kB
bounce:108kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Mar 13 06:30:35 node1 kernel: [902670.903085] lowmem_reserve[]: 0 0 25321 25321
Mar 13 06:30:35 node1 kernel: [902670.903091] HighMem free:706596kB min:512kB low:3920kB high:7332kB active_anon:544944kB inactive_anon:142308kB
active_file:692736kB inactive_file:1140028kB unevictable:16332kB isolated(anon):0kB isolated(file):0kB present:3241132kB mlocked:16332kB dirty:6604kB
writeback:0kB mapped:94524kB shmem:46144kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:9072kB unstable:0kB bounce:0kB
writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Mar 13 06:30:35 node1 kernel: [902670.903095] lowmem_reserve[]: 0 0 0 0
Mar 13 06:30:35 node1 kernel: [902670.903097] DMA: 15*4kB 15*8kB 9*16kB 22*32kB 7*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 1*2048kB 0*4096kB = 3524kB
Mar 13 06:30:35 node1 kernel: [902670.903104] Normal: 3507*4kB 3160*8kB 37*16kB 8*32kB 2*64kB 1*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 40412kB
Mar 13 06:30:35 node1 kernel: [902670.903110] HighMem: 77*4kB 56*8kB 23*16kB 12*32kB 2061*64kB 2550*128kB 704*256kB 104*512kB 13*1024kB 0*2048kB
0*4096kB = 706596kB
Mar 13 06:30:35 node1 kernel: [902670.903117] 612114 total pagecache pages
Mar 13 06:30:35 node1 kernel: [902670.903118] 160 pages in swap cache
Mar 13 06:30:35 node1 kernel: [902670.903120] Swap cache stats: add 21929, delete 21769, find 243894/245286
Mar 13 06:30:35 node1 kernel: [902670.903121] Free swap  = 4880236kB
Mar 13 06:30:35 node1 kernel: [902670.903122] Total swap = 4883640kB
Mar 13 06:30:35 node1 kernel: [902670.912442] 1245184 pages RAM
Mar 13 06:30:35 node1 kernel: [902670.912443] 1017858 pages HighMem
Mar 13 06:30:35 node1 kernel: [902670.912444] 210949 pages reserved
Mar 13 06:30:35 node1 kernel: [902670.912445] 327375 pages shared
Mar 13 06:30:35 node1 kernel: [902670.912446] 642252 pages non-shared
Mar 13 06:30:35 node1 kernel: [902670.912462] drbd1_receiver: page allocation failure. order:4, mode:0x4020
Mar 13 06:30:35 node1 kernel: [902670.912465] Pid: 25245, comm: drbd1_receiver Not tainted 2.6.32-5-vserver-686-bigmem #1
Mar 13 06:30:35 node1 kernel: [902670.912467] Call Trace:
Mar 13 06:30:35 node1 kernel: [902670.912472]  [<c109ad8c>] ? __alloc_pages_nodemask+0x484/0x4d9
Mar 13 06:30:35 node1 kernel: [902670.912475]  [<c109aded>] ? __get_free_pages+0xc/0x17
Mar 13 06:30:35 node1 kernel: [902670.912478]  [<c10bf261>] ? __kmalloc+0x30/0x128
Mar 13 06:30:35 node1 kernel: [902670.912487]  [<c11ee2f4>] ? pskb_expand_head+0x4f/0x157
Mar 13 06:30:35 node1 kernel: [902670.912490]  [<c11ee74b>] ? __pskb_pull_tail+0x40/0x200
Mar 13 06:30:35 node1 kernel: [902670.912492]  [<c11eb077>] ? sock_wfree+0x17/0x4b
Mar 13 06:30:35 node1 kernel: [902670.912495]  [<c11f555e>] ? dev_queue_xmit+0xe4/0x38e
Mar 13 06:30:35 node1 kernel: [902670.912497]  [<c1216cea>] ? ip_finish_output2+0x187/0x1c2
Mar 13 06:30:35 node1 kernel: [902670.912500]  [<c12161ed>] ? ip_local_out+0x15/0x17
Mar 13 06:30:35 node1 kernel: [902670.912502]  [<c12169cc>] ? ip_queue_xmit+0x31d/0x378
Mar 13 06:30:35 node1 kernel: [902670.912505]  [<c1224fdf>] ? tcp_established_options+0x1d/0x8b
Mar 13 06:30:35 node1 kernel: [902670.912507]  [<c1225c52>] ? tcp_transmit_skb+0x595/0x5cc
Mar 13 06:30:35 node1 kernel: [902670.912510]  [<c1227cb5>] ? tcp_write_xmit+0x7a3/0x874
Mar 13 06:30:35 node1 kernel: [902670.912512]  [<c12236e4>] ? tcp_ack+0x1611/0x1802
Mar 13 06:30:35 node1 kernel: [902670.912514]  [<c1224fdf>] ? tcp_established_options+0x1d/0x8b
Mar 13 06:30:35 node1 kernel: [902670.912516]  [<c12252a3>] ? tcp_current_mss+0x38/0x53
Mar 13 06:30:35 node1 kernel: [902670.912519]  [<c1227dcc>] ? __tcp_push_pending_frames+0x1e/0x50
Mar 13 06:30:35 node1 kernel: [902670.912521]  [<c12238f0>] ? tcp_data_snd_check+0x1b/0xd2
Mar 13 06:30:35 node1 kernel: [902670.912523]  [<c1224493>] ? tcp_rcv_established+0x5d6/0x62c
Mar 13 06:30:35 node1 kernel: [902670.912526]  [<c128afe9>] ? schedule_timeout+0x20/0xb0
Mar 13 06:30:35 node1 kernel: [902670.912529]  [<c122a9ae>] ? tcp_v4_do_rcv+0x15f/0x2cf
Mar 13 06:30:35 node1 kernel: [902670.912531]  [<c11eabce>] ? sk_wait_data+0x99/0xa2
Mar 13 06:30:35 node1 kernel: [902670.912534]  [<c121bf64>] ? tcp_prequeue_process+0x57/0x6b
Mar 13 06:30:35 node1 kernel: [902670.912536]  [<c121cf66>] ? tcp_recvmsg+0x441/0x8e5
Mar 13 06:30:35 node1 kernel: [902670.912538]  [<c11e9cdf>] ? sock_common_recvmsg+0x2f/0x45
Mar 13 06:30:35 node1 kernel: [902670.912541]  [<c11e9cdf>] ? sock_common_recvmsg+0x2f/0x45
Mar 13 06:30:35 node1 kernel: [902670.912544]  [<c11e7fc3>] ? __sock_recvmsg+0x50/0xbb
Mar 13 06:30:35 node1 kernel: [902670.912546]  [<c11e855d>] ? sock_recvmsg+0x9d/0xb6
Mar 13 06:30:35 node1 kernel: [902670.912549]  [<c104b08a>] ? autoremove_wake_function+0x0/0x2d
Mar 13 06:30:35 node1 kernel: [902670.912553]  [<c113c658>] ? __make_request+0x364/0x3d9
Mar 13 06:30:35 node1 kernel: [902670.912558]  [<f9067ddf>] ? drbd_recv+0x62/0x106 [drbd]
Mar 13 06:30:35 node1 kernel: [902670.912563]  [<f9067e95>] ? drbd_recv_header+0x12/0xa5 [drbd]
Mar 13 06:30:35 node1 kernel: [902670.912568]  [<f906e669>] ? drbdd_init+0xf3/0x245 [drbd]
Mar 13 06:30:35 node1 kernel: [902670.912572]  [<f9079bf7>] ? drbd_thread_setup+0x1f/0xcc [drbd]
Mar 13 06:30:35 node1 kernel: [902670.912577]  [<f9079bd8>] ? drbd_thread_setup+0x0/0xcc [drbd]
Mar 13 06:30:35 node1 kernel: [902670.912579]  [<c104ae58>] ? kthread+0x61/0x66
Mar 13 06:30:35 node1 kernel: [902670.912581]  [<c104adf7>] ? kthread+0x0/0x66
Mar 13 06:30:35 node1 kernel: [902670.912584]  [<c1008d87>] ? kernel_thread_helper+0x7/0x10
Mar 13 06:30:35 node1 kernel: [902670.912585] Mem-Info:
Mar 13 06:30:35 node1 kernel: [902670.912587] DMA per-cpu:
Mar 13 06:30:35 node1 kernel: [902670.912588] CPU    0: hi:    0, btch:   1 usd:   0
Mar 13 06:30:35 node1 kernel: [902670.912590] CPU    1: hi:    0, btch:   1 usd:   0
Mar 13 06:30:35 node1 kernel: [902670.912591] Normal per-cpu:
Mar 13 06:30:35 node1 kernel: [902670.912592] CPU    0: hi:  186, btch:  31 usd: 173
Mar 13 06:30:35 node1 kernel: [902670.912594] CPU    1: hi:  186, btch:  31 usd: 154
Mar 13 06:30:35 node1 kernel: [902670.912595] HighMem per-cpu:
Mar 13 06:30:35 node1 kernel: [902670.912596] CPU    0: hi:  186, btch:  31 usd:  47
Mar 13 06:30:35 node1 kernel: [902670.912598] CPU    1: hi:  186, btch:  31 usd:  12
Mar 13 06:30:35 node1 kernel: [902670.912601] active_anon:137306 inactive_anon:39520 isolated_anon:0
Mar 13 06:30:35 node1 kernel: [902670.912602]  active_file:223625 inactive_file:375450 isolated_file:32
Mar 13 06:30:35 node1 kernel: [902670.912603]  unevictable:4083 dirty:1708 writeback:0 unstable:0
Mar 13 06:30:35 node1 kernel: [902670.912604]  free:187603 slab_reclaimable:50120 slab_unreclaimable:3975
Mar 13 06:30:35 node1 kernel: [902670.912604]  mapped:23637 shmem:11538 pagetables:2294 bounce:27
Mar 13 06:30:35 node1 kernel: [902670.912609] DMA free:3524kB min:64kB low:80kB high:96kB active_anon:0kB inactive_anon:0kB active_file:2728kB
inactive_file:6180kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15868kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB
slab_reclaimable:3512kB slab_unreclaimable:8kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0
all_unreclaimable? no
Mar 13 06:30:35 node1 kernel: [902670.912613] lowmem_reserve[]: 0 865 4030 4030
Mar 13 06:30:35 node1 kernel: [902670.912619] Normal free:40292kB min:3728kB low:4660kB high:5592kB active_anon:4280kB inactive_anon:15772kB
active_file:199036kB inactive_file:355444kB unevictable:0kB isolated(anon):0kB isolated(file):128kB present:885944kB mlocked:0kB dirty:228kB
writeback:0kB mapped:24kB shmem:8kB slab_reclaimable:196968kB slab_unreclaimable:15892kB kernel_stack:2608kB pagetables:104kB unstable:0kB
bounce:108kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Mar 13 06:30:35 node1 kernel: [902670.912623] lowmem_reserve[]: 0 0 25321 25321
Mar 13 06:30:35 node1 kernel: [902670.912629] HighMem free:706596kB min:512kB low:3920kB high:7332kB active_anon:544944kB inactive_anon:142308kB
active_file:692736kB inactive_file:1140176kB unevictable:16332kB isolated(anon):0kB isolated(file):0kB present:3241132kB mlocked:16332kB dirty:6604kB
writeback:0kB mapped:94524kB shmem:46144kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:9072kB unstable:0kB bounce:0kB
writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Mar 13 06:30:35 node1 kernel: [902670.912633] lowmem_reserve[]: 0 0 0 0
Mar 13 06:30:35 node1 kernel: [902670.912636] DMA: 15*4kB 15*8kB 9*16kB 22*32kB 7*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 1*2048kB 0*4096kB = 3524kB
Mar 13 06:30:35 node1 kernel: [902670.912642] Normal: 3507*4kB 3160*8kB 37*16kB 8*32kB 2*64kB 1*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 40412kB
Mar 13 06:30:35 node1 kernel: [902670.912648] HighMem: 77*4kB 56*8kB 23*16kB 12*32kB 2061*64kB 2550*128kB 704*256kB 104*512kB 13*1024kB 0*2048kB
0*4096kB = 706596kB
Mar 13 06:30:35 node1 kernel: [902670.912655] 612114 total pagecache pages
Mar 13 06:30:35 node1 kernel: [902670.912656] 160 pages in swap cache
Mar 13 06:30:35 node1 kernel: [902670.912658] Swap cache stats: add 21929, delete 21769, find 243894/245286
Mar 13 06:30:35 node1 kernel: [902670.912659] Free swap  = 4880236kB
Mar 13 06:30:35 node1 kernel: [902670.912660] Total swap = 4883640kB
Mar 13 06:30:35 node1 kernel: [902670.922012] 1245184 pages RAM
Mar 13 06:30:35 node1 kernel: [902670.922014] 1017858 pages HighMem
Mar 13 06:30:35 node1 kernel: [902670.922015] 210949 pages reserved
Mar 13 06:30:35 node1 kernel: [902670.922016] 327379 pages shared
Mar 13 06:30:35 node1 kernel: [902670.922017] 642254 pages non-shared
Mar 13 06:30:35 node1 kernel: [902670.922033] drbd1_receiver: page allocation failure. order:4, mode:0x4020
Mar 13 06:30:35 node1 kernel: [902670.922036] Pid: 25245, comm: drbd1_receiver Not tainted 2.6.32-5-vserver-686-bigmem #1
Mar 13 06:30:35 node1 kernel: [902670.922037] Call Trace:
Mar 13 06:30:35 node1 kernel: [902670.922043]  [<c109ad8c>] ? __alloc_pages_nodemask+0x484/0x4d9
Mar 13 06:30:35 node1 kernel: [902670.922045]  [<c109aded>] ? __get_free_pages+0xc/0x17
Mar 13 06:30:35 node1 kernel: [902670.922048]  [<c10bf261>] ? __kmalloc+0x30/0x128
Mar 13 06:30:35 node1 kernel: [902670.922057]  [<c11ee2f4>] ? pskb_expand_head+0x4f/0x157
Mar 13 06:30:35 node1 kernel: [902670.922060]  [<c11ee74b>] ? __pskb_pull_tail+0x40/0x200
Mar 13 06:30:35 node1 kernel: [902670.922062]  [<c11eb077>] ? sock_wfree+0x17/0x4b
Mar 13 06:30:35 node1 kernel: [902670.922065]  [<c11f555e>] ? dev_queue_xmit+0xe4/0x38e
Mar 13 06:30:35 node1 kernel: [902670.922067]  [<c1216cea>] ? ip_finish_output2+0x187/0x1c2
Mar 13 06:30:35 node1 kernel: [902670.922070]  [<c12161ed>] ? ip_local_out+0x15/0x17
Mar 13 06:30:35 node1 kernel: [902670.922072]  [<c12169cc>] ? ip_queue_xmit+0x31d/0x378
Mar 13 06:30:35 node1 kernel: [902670.922075]  [<c1224fdf>] ? tcp_established_options+0x1d/0x8b
Mar 13 06:30:35 node1 kernel: [902670.922077]  [<c1225c52>] ? tcp_transmit_skb+0x595/0x5cc
Mar 13 06:30:35 node1 kernel: [902670.922079]  [<c1227cb5>] ? tcp_write_xmit+0x7a3/0x874
Mar 13 06:30:35 node1 kernel: [902670.922082]  [<c12236e4>] ? tcp_ack+0x1611/0x1802
Mar 13 06:30:35 node1 kernel: [902670.922084]  [<c1224fdf>] ? tcp_established_options+0x1d/0x8b
Mar 13 06:30:35 node1 kernel: [902670.922086]  [<c12252a3>] ? tcp_current_mss+0x38/0x53
Mar 13 06:30:35 node1 kernel: [902670.922088]  [<c1227dcc>] ? __tcp_push_pending_frames+0x1e/0x50
Mar 13 06:30:35 node1 kernel: [902670.922091]  [<c12238f0>] ? tcp_data_snd_check+0x1b/0xd2
Mar 13 06:30:35 node1 kernel: [902670.922093]  [<c1224493>] ? tcp_rcv_established+0x5d6/0x62c
Mar 13 06:30:35 node1 kernel: [902670.922096]  [<c128afe9>] ? schedule_timeout+0x20/0xb0
Mar 13 06:30:35 node1 kernel: [902670.922098]  [<c122a9ae>] ? tcp_v4_do_rcv+0x15f/0x2cf
Mar 13 06:30:35 node1 kernel: [902670.922101]  [<c11eabce>] ? sk_wait_data+0x99/0xa2
Mar 13 06:30:35 node1 kernel: [902670.922103]  [<c121bf64>] ? tcp_prequeue_process+0x57/0x6b
Mar 13 06:30:35 node1 kernel: [902670.922106]  [<c121cf66>] ? tcp_recvmsg+0x441/0x8e5
Mar 13 06:30:35 node1 kernel: [902670.922108]  [<c11e9cdf>] ? sock_common_recvmsg+0x2f/0x45
Mar 13 06:30:35 node1 kernel: [902670.922110]  [<c11e9cdf>] ? sock_common_recvmsg+0x2f/0x45
Mar 13 06:30:35 node1 kernel: [902670.922114]  [<c11e7fc3>] ? __sock_recvmsg+0x50/0xbb
Mar 13 06:30:35 node1 kernel: [902670.922116]  [<c11e855d>] ? sock_recvmsg+0x9d/0xb6
Mar 13 06:30:35 node1 kernel: [902670.922120]  [<c104b08a>] ? autoremove_wake_function+0x0/0x2d
Mar 13 06:30:35 node1 kernel: [902670.922123]  [<c113c658>] ? __make_request+0x364/0x3d9
Mar 13 06:30:35 node1 kernel: [902670.922129]  [<f9067ddf>] ? drbd_recv+0x62/0x106 [drbd]
Mar 13 06:30:35 node1 kernel: [902670.922134]  [<f9067e95>] ? drbd_recv_header+0x12/0xa5 [drbd]
Mar 13 06:30:35 node1 kernel: [902670.922138]  [<f906e669>] ? drbdd_init+0xf3/0x245 [drbd]
Mar 13 06:30:35 node1 kernel: [902670.922143]  [<f9079bf7>] ? drbd_thread_setup+0x1f/0xcc [drbd]
Mar 13 06:30:35 node1 kernel: [902670.922147]  [<f9079bd8>] ? drbd_thread_setup+0x0/0xcc [drbd]
Mar 13 06:30:35 node1 kernel: [902670.922150]  [<c104ae58>] ? kthread+0x61/0x66
Mar 13 06:30:35 node1 kernel: [902670.922152]  [<c104adf7>] ? kthread+0x0/0x66
Mar 13 06:30:35 node1 kernel: [902670.922155]  [<c1008d87>] ? kernel_thread_helper+0x7/0x10
Mar 13 06:30:35 node1 kernel: [902670.922156] Mem-Info:
Mar 13 06:30:35 node1 kernel: [902670.922157] DMA per-cpu:
Mar 13 06:30:35 node1 kernel: [902670.922159] CPU    0: hi:    0, btch:   1 usd:   0
Mar 13 06:30:35 node1 kernel: [902670.922160] CPU    1: hi:    0, btch:   1 usd:   0
Mar 13 06:30:35 node1 kernel: [902670.922161] Normal per-cpu:
Mar 13 06:30:35 node1 kernel: [902670.922163] CPU    0: hi:  186, btch:  31 usd: 173
Mar 13 06:30:35 node1 kernel: [902670.922164] CPU    1: hi:  186, btch:  31 usd: 154
Mar 13 06:30:35 node1 kernel: [902670.922166] HighMem per-cpu:
Mar 13 06:30:35 node1 kernel: [902670.922167] CPU    0: hi:  186, btch:  31 usd:  47
Mar 13 06:30:35 node1 kernel: [902670.922168] CPU    1: hi:  186, btch:  31 usd:  35
Mar 13 06:30:35 node1 kernel: [902670.922172] active_anon:137306 inactive_anon:39520 isolated_anon:0
Mar 13 06:30:35 node1 kernel: [902670.922173]  active_file:223625 inactive_file:375450 isolated_file:32
Mar 13 06:30:35 node1 kernel: [902670.922173]  unevictable:4083 dirty:1708 writeback:0 unstable:0
Mar 13 06:30:35 node1 kernel: [902670.922174]  free:187572 slab_reclaimable:50120 slab_unreclaimable:3975
Mar 13 06:30:35 node1 kernel: [902670.922175]  mapped:23637 shmem:11538 pagetables:2294 bounce:27
Mar 13 06:30:35 node1 kernel: [902670.922180] DMA free:3524kB min:64kB low:80kB high:96kB active_anon:0kB inactive_anon:0kB active_file:2728kB
inactive_file:6180kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15868kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB
slab_reclaimable:3512kB slab_unreclaimable:8kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0
all_unreclaimable? no
Mar 13 06:30:35 node1 kernel: [902670.922184] lowmem_reserve[]: 0 865 4030 4030
Mar 13 06:30:35 node1 kernel: [902670.922190] Normal free:40292kB min:3728kB low:4660kB high:5592kB active_anon:4280kB inactive_anon:15772kB
active_file:199036kB inactive_file:355444kB unevictable:0kB isolated(anon):0kB isolated(file):128kB present:885944kB mlocked:0kB dirty:228kB
writeback:0kB mapped:24kB shmem:8kB slab_reclaimable:196968kB slab_unreclaimable:15892kB kernel_stack:2608kB pagetables:104kB unstable:0kB
bounce:108kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Mar 13 06:30:35 node1 kernel: [902670.922194] lowmem_reserve[]: 0 0 25321 25321
Mar 13 06:30:35 node1 kernel: [902670.922199] HighMem free:706472kB min:512kB low:3920kB high:7332kB active_anon:544944kB inactive_anon:142308kB
active_file:692736kB inactive_file:1140176kB unevictable:16332kB isolated(anon):0kB isolated(file):0kB present:3241132kB mlocked:16332kB dirty:6604kB
writeback:0kB mapped:94524kB shmem:46144kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:9072kB unstable:0kB bounce:0kB
writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Mar 13 06:30:35 node1 kernel: [902670.922204] lowmem_reserve[]: 0 0 0 0
Mar 13 06:30:35 node1 kernel: [902670.922206] DMA: 15*4kB 15*8kB 9*16kB 22*32kB 7*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 1*2048kB 0*4096kB = 3524kB
Mar 13 06:30:35 node1 kernel: [902670.922213] Normal: 3507*4kB 3160*8kB 37*16kB 8*32kB 2*64kB 1*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 40412kB
Mar 13 06:30:35 node1 kernel: [902670.922219] HighMem: 78*4kB 56*8kB 23*16kB 12*32kB 2059*64kB 2550*128kB 704*256kB 104*512kB 13*1024kB 0*2048kB
0*4096kB = 706472kB
Mar 13 06:30:35 node1 kernel: [902670.922226] 612114 total pagecache pages
Mar 13 06:30:35 node1 kernel: [902670.922227] 160 pages in swap cache
Mar 13 06:30:35 node1 kernel: [902670.922228] Swap cache stats: add 21929, delete 21769, find 243894/245286
Mar 13 06:30:35 node1 kernel: [902670.922230] Free swap  = 4880236kB
Mar 13 06:30:35 node1 kernel: [902670.922231] Total swap = 4883640kB
Mar 13 06:30:35 node1 kernel: [902670.931613] 1245184 pages RAM
Mar 13 06:30:35 node1 kernel: [902670.931615] 1017858 pages HighMem
Mar 13 06:30:35 node1 kernel: [902670.931616] 210949 pages reserved
Mar 13 06:30:35 node1 kernel: [902670.931617] 327383 pages shared
Mar 13 06:30:35 node1 kernel: [902670.931618] 642256 pages non-shared
Mar 13 06:30:35 node1 kernel: [902670.931634] drbd1_receiver: page allocation failure. order:4, mode:0x4020
Mar 13 06:30:35 node1 kernel: [902670.931636] Pid: 25245, comm: drbd1_receiver Not tainted 2.6.32-5-vserver-686-bigmem #1
Mar 13 06:30:35 node1 kernel: [902670.931638] Call Trace:
Mar 13 06:30:35 node1 kernel: [902670.931643]  [<c109ad8c>] ? __alloc_pages_nodemask+0x484/0x4d9
Mar 13 06:30:35 node1 kernel: [902670.931646]  [<c109aded>] ? __get_free_pages+0xc/0x17
Mar 13 06:30:35 node1 kernel: [902670.931649]  [<c10bf261>] ? __kmalloc+0x30/0x128
Mar 13 06:30:35 node1 kernel: [902670.931658]  [<c11ee2f4>] ? pskb_expand_head+0x4f/0x157
Mar 13 06:30:35 node1 kernel: [902670.931660]  [<c11ee74b>] ? __pskb_pull_tail+0x40/0x200
Mar 13 06:30:35 node1 kernel: [902670.931663]  [<c11eb077>] ? sock_wfree+0x17/0x4b
Mar 13 06:30:35 node1 kernel: [902670.931665]  [<c11f555e>] ? dev_queue_xmit+0xe4/0x38e
Mar 13 06:30:35 node1 kernel: [902670.931668]  [<c1216cea>] ? ip_finish_output2+0x187/0x1c2
Mar 13 06:30:35 node1 kernel: [902670.931671]  [<c12161ed>] ? ip_local_out+0x15/0x17
Mar 13 06:30:35 node1 kernel: [902670.931673]  [<c12169cc>] ? ip_queue_xmit+0x31d/0x378
Mar 13 06:30:35 node1 kernel: [902670.931676]  [<c1224fdf>] ? tcp_established_options+0x1d/0x8b
Mar 13 06:30:35 node1 kernel: [902670.931678]  [<c1225c52>] ? tcp_transmit_skb+0x595/0x5cc
Mar 13 06:30:35 node1 kernel: [902670.931681]  [<c1227cb5>] ? tcp_write_xmit+0x7a3/0x874
Mar 13 06:30:35 node1 kernel: [902670.931683]  [<c12236e4>] ? tcp_ack+0x1611/0x1802
Mar 13 06:30:35 node1 kernel: [902670.931685]  [<c1224fdf>] ? tcp_established_options+0x1d/0x8b
Mar 13 06:30:35 node1 kernel: [902670.931687]  [<c12252a3>] ? tcp_current_mss+0x38/0x53
Mar 13 06:30:35 node1 kernel: [902670.931689]  [<c1227dcc>] ? __tcp_push_pending_frames+0x1e/0x50
Mar 13 06:30:35 node1 kernel: [902670.931692]  [<c12238f0>] ? tcp_data_snd_check+0x1b/0xd2
Mar 13 06:30:35 node1 kernel: [902670.931694]  [<c1224493>] ? tcp_rcv_established+0x5d6/0x62c
Mar 13 06:30:35 node1 kernel: [902670.931697]  [<c128afe9>] ? schedule_timeout+0x20/0xb0
Mar 13 06:30:35 node1 kernel: [902670.931700]  [<c122a9ae>] ? tcp_v4_do_rcv+0x15f/0x2cf
Mar 13 06:30:35 node1 kernel: [902670.931702]  [<c11eabce>] ? sk_wait_data+0x99/0xa2
Mar 13 06:30:35 node1 kernel: [902670.931705]  [<c121bf64>] ? tcp_prequeue_process+0x57/0x6b
Mar 13 06:30:35 node1 kernel: [902670.931707]  [<c121cf66>] ? tcp_recvmsg+0x441/0x8e5
Mar 13 06:30:35 node1 kernel: [902670.931709]  [<c11e9cdf>] ? sock_common_recvmsg+0x2f/0x45
Mar 13 06:30:35 node1 kernel: [902670.931712]  [<c11e9cdf>] ? sock_common_recvmsg+0x2f/0x45
Mar 13 06:30:35 node1 kernel: [902670.931715]  [<c11e7fc3>] ? __sock_recvmsg+0x50/0xbb
Mar 13 06:30:35 node1 kernel: [902670.931717]  [<c11e855d>] ? sock_recvmsg+0x9d/0xb6
Mar 13 06:30:35 node1 kernel: [902670.931720]  [<c104b08a>] ? autoremove_wake_function+0x0/0x2d
Mar 13 06:30:35 node1 kernel: [902670.931724]  [<c113c658>] ? __make_request+0x364/0x3d9
Mar 13 06:30:35 node1 kernel: [902670.931729]  [<f9067ddf>] ? drbd_recv+0x62/0x106 [drbd]
Mar 13 06:30:35 node1 kernel: [902670.931734]  [<f9067e95>] ? drbd_recv_header+0x12/0xa5 [drbd]
Mar 13 06:30:35 node1 kernel: [902670.931739]  [<f906e669>] ? drbdd_init+0xf3/0x245 [drbd]
Mar 13 06:30:35 node1 kernel: [902670.931743]  [<f9079bf7>] ? drbd_thread_setup+0x1f/0xcc [drbd]
Mar 13 06:30:35 node1 kernel: [902670.931748]  [<f9079bd8>] ? drbd_thread_setup+0x0/0xcc [drbd]
Mar 13 06:30:35 node1 kernel: [902670.931750]  [<c104ae58>] ? kthread+0x61/0x66
Mar 13 06:30:35 node1 kernel: [902670.931752]  [<c104adf7>] ? kthread+0x0/0x66
Mar 13 06:30:35 node1 kernel: [902670.931755]  [<c1008d87>] ? kernel_thread_helper+0x7/0x10
Mar 13 06:30:35 node1 kernel: [902670.931756] Mem-Info:
Mar 13 06:30:35 node1 kernel: [902670.931757] DMA per-cpu:
Mar 13 06:30:35 node1 kernel: [902670.931759] CPU    0: hi:    0, btch:   1 usd:   0
Mar 13 06:30:35 node1 kernel: [902670.931760] CPU    1: hi:    0, btch:   1 usd:   0
Mar 13 06:30:35 node1 kernel: [902670.931762] Normal per-cpu:
Mar 13 06:30:35 node1 kernel: [902670.931763] CPU    0: hi:  186, btch:  31 usd: 173
Mar 13 06:30:35 node1 kernel: [902670.931764] CPU    1: hi:  186, btch:  31 usd: 154
Mar 13 06:30:35 node1 kernel: [902670.931766] HighMem per-cpu:
Mar 13 06:30:35 node1 kernel: [902670.931767] CPU    0: hi:  186, btch:  31 usd:  47
Mar 13 06:30:35 node1 kernel: [902670.931769] CPU    1: hi:  186, btch:  31 usd:  27
Mar 13 06:30:35 node1 kernel: [902670.931772] active_anon:137306 inactive_anon:39520 isolated_anon:0
Mar 13 06:30:35 node1 kernel: [902670.931773]  active_file:223625 inactive_file:375450 isolated_file:32
Mar 13 06:30:35 node1 kernel: [902670.931774]  unevictable:4083 dirty:1708 writeback:0 unstable:0
Mar 13 06:30:35 node1 kernel: [902670.931774]  free:187572 slab_reclaimable:50120 slab_unreclaimable:3975
Mar 13 06:30:35 node1 kernel: [902670.931775]  mapped:23637 shmem:11538 pagetables:2294 bounce:27
Mar 13 06:30:35 node1 kernel: [902670.931780] DMA free:3524kB min:64kB low:80kB high:96kB active_anon:0kB inactive_anon:0kB active_file:2728kB
inactive_file:6180kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15868kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB
slab_reclaimable:3512kB slab_unreclaimable:8kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0
all_unreclaimable? no
Mar 13 06:30:35 node1 kernel: [902670.931784] lowmem_reserve[]: 0 865 4030 4030
Mar 13 06:30:35 node1 kernel: [902670.931790] Normal free:40292kB min:3728kB low:4660kB high:5592kB active_anon:4280kB inactive_anon:15772kB
active_file:199036kB inactive_file:355444kB unevictable:0kB isolated(anon):0kB isolated(file):128kB present:885944kB mlocked:0kB dirty:228kB
writeback:0kB mapped:24kB shmem:8kB slab_reclaimable:196968kB slab_unreclaimable:15892kB kernel_stack:2608kB pagetables:104kB unstable:0kB
bounce:108kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Mar 13 06:30:35 node1 kernel: [902670.931794] lowmem_reserve[]: 0 0 25321 25321
Mar 13 06:30:35 node1 kernel: [902670.931799] HighMem free:706472kB min:512kB low:3920kB high:7332kB active_anon:544944kB inactive_anon:142308kB
active_file:692736kB inactive_file:1140176kB unevictable:16332kB isolated(anon):0kB isolated(file):0kB present:3241132kB mlocked:16332kB dirty:6604kB
writeback:0kB mapped:94524kB shmem:46144kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:9072kB unstable:0kB bounce:0kB
writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Mar 13 06:30:35 node1 kernel: [902670.931804] lowmem_reserve[]: 0 0 0 0
Mar 13 06:30:35 node1 kernel: [902670.931806] DMA: 15*4kB 15*8kB 9*16kB 22*32kB 7*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 1*2048kB 0*4096kB = 3524kB
Mar 13 06:30:35 node1 kernel: [902670.931813] Normal: 3507*4kB 3160*8kB 37*16kB 8*32kB 2*64kB 1*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 40412kB
Mar 13 06:30:35 node1 kernel: [902670.931819] HighMem: 78*4kB 56*8kB 23*16kB 12*32kB 2059*64kB 2550*128kB 704*256kB 104*512kB 13*1024kB 0*2048kB
0*4096kB = 706472kB
Mar 13 06:30:35 node1 kernel: [902670.931826] 612114 total pagecache pages
Mar 13 06:30:35 node1 kernel: [902670.931827] 160 pages in swap cache
Mar 13 06:30:35 node1 kernel: [902670.931828] Swap cache stats: add 21929, delete 21769, find 243894/245286
Mar 13 06:30:35 node1 kernel: [902670.931830] Free swap  = 4880236kB
Mar 13 06:30:35 node1 kernel: [902670.931831] Total swap = 4883640kB
Mar 13 06:30:35 node1 kernel: [902670.941170] 1245184 pages RAM
Mar 13 06:30:35 node1 kernel: [902670.941171] 1017858 pages HighMem
Mar 13 06:30:35 node1 kernel: [902670.941172] 210949 pages reserved
Mar 13 06:30:35 node1 kernel: [902670.941173] 327394 pages shared
Mar 13 06:30:35 node1 kernel: [902670.941174] 642258 pages non-shared
Mar 13 06:30:35 node1 kernel: [902670.941986] drbd1_receiver: page allocation failure. order:4, mode:0x4020
Mar 13 06:30:35 node1 kernel: [902670.941989] Pid: 25245, comm: drbd1_receiver Not tainted 2.6.32-5-vserver-686-bigmem #1
Mar 13 06:30:35 node1 kernel: [902670.941990] Call Trace:
Mar 13 06:30:35 node1 kernel: [902670.941997]  [<c109ad8c>] ? __alloc_pages_nodemask+0x484/0x4d9
Mar 13 06:30:35 node1 kernel: [902670.941999]  [<c109aded>] ? __get_free_pages+0xc/0x17
Mar 13 06:30:35 node1 kernel: [902670.942002]  [<c10bf261>] ? __kmalloc+0x30/0x128
Mar 13 06:30:35 node1 kernel: [902670.942013]  [<c11ee2f4>] ? pskb_expand_head+0x4f/0x157
Mar 13 06:30:35 node1 kernel: [902670.942015]  [<c11ee74b>] ? __pskb_pull_tail+0x40/0x200
Mar 13 06:30:35 node1 kernel: [902670.942018]  [<c11f555e>] ? dev_queue_xmit+0xe4/0x38e
Mar 13 06:30:35 node1 kernel: [902670.942021]  [<c1216cea>] ? ip_finish_output2+0x187/0x1c2
Mar 13 06:30:35 node1 kernel: [902670.942024]  [<c12161ed>] ? ip_local_out+0x15/0x17
Mar 13 06:30:35 node1 kernel: [902670.942026]  [<c12169cc>] ? ip_queue_xmit+0x31d/0x378
Mar 13 06:30:35 node1 kernel: [902670.942033]  [<f81b7696>] ? scsi_init_sgtable+0x2d/0x46 [scsi_mod]
Mar 13 06:30:35 node1 kernel: [902670.942037]  [<c1225c52>] ? tcp_transmit_skb+0x595/0x5cc
Mar 13 06:30:35 node1 kernel: [902670.942039]  [<c1227cb5>] ? tcp_write_xmit+0x7a3/0x874
Mar 13 06:30:35 node1 kernel: [902670.942041]  [<c12236e4>] ? tcp_ack+0x1611/0x1802
Mar 13 06:30:35 node1 kernel: [902670.942044]  [<c1224fdf>] ? tcp_established_options+0x1d/0x8b
Mar 13 06:30:35 node1 kernel: [902670.942046]  [<c12252a3>] ? tcp_current_mss+0x38/0x53
Mar 13 06:30:35 node1 kernel: [902670.942049]  [<c1227dcc>] ? __tcp_push_pending_frames+0x1e/0x50
Mar 13 06:30:35 node1 kernel: [902670.942051]  [<c12238f0>] ? tcp_data_snd_check+0x1b/0xd2
Mar 13 06:30:35 node1 kernel: [902670.942053]  [<c1224493>] ? tcp_rcv_established+0x5d6/0x62c
Mar 13 06:30:35 node1 kernel: [902670.942056]  [<c122a9ae>] ? tcp_v4_do_rcv+0x15f/0x2cf
Mar 13 06:30:35 node1 kernel: [902670.942058]  [<c11f2e36>] ? net_tx_action+0xa9/0xf9
Mar 13 06:30:35 node1 kernel: [902670.942060]  [<c11ea5ff>] ? release_sock+0x38/0x89
Mar 13 06:30:35 node1 kernel: [902670.942063]  [<c11eab8a>] ? sk_wait_data+0x55/0xa2
Mar 13 06:30:35 node1 kernel: [902670.942066]  [<c104b08a>] ? autoremove_wake_function+0x0/0x2d
Mar 13 06:30:35 node1 kernel: [902670.942068]  [<c121cf09>] ? tcp_recvmsg+0x3e4/0x8e5
Mar 13 06:30:35 node1 kernel: [902670.942071]  [<c11e9cdf>] ? sock_common_recvmsg+0x2f/0x45
Mar 13 06:30:35 node1 kernel: [902670.942073]  [<c11e9cdf>] ? sock_common_recvmsg+0x2f/0x45
Mar 13 06:30:35 node1 kernel: [902670.942076]  [<c11e7fc3>] ? __sock_recvmsg+0x50/0xbb
Mar 13 06:30:35 node1 kernel: [902670.942078]  [<c11e855d>] ? sock_recvmsg+0x9d/0xb6
Mar 13 06:30:35 node1 kernel: [902670.942081]  [<c104b08a>] ? autoremove_wake_function+0x0/0x2d
Mar 13 06:30:35 node1 kernel: [902670.942084]  [<c113c658>] ? __make_request+0x364/0x3d9
Mar 13 06:30:35 node1 kernel: [902670.942089]  [<f9067ddf>] ? drbd_recv+0x62/0x106 [drbd]
Mar 13 06:30:35 node1 kernel: [902670.942094]  [<f9067e95>] ? drbd_recv_header+0x12/0xa5 [drbd]
Mar 13 06:30:35 node1 kernel: [902670.942099]  [<f906e669>] ? drbdd_init+0xf3/0x245 [drbd]
Mar 13 06:30:35 node1 kernel: [902670.942103]  [<f9079bf7>] ? drbd_thread_setup+0x1f/0xcc [drbd]
Mar 13 06:30:35 node1 kernel: [902670.942108]  [<f9079bd8>] ? drbd_thread_setup+0x0/0xcc [drbd]
Mar 13 06:30:35 node1 kernel: [902670.942110]  [<c104ae58>] ? kthread+0x61/0x66
Mar 13 06:30:35 node1 kernel: [902670.942112]  [<c104adf7>] ? kthread+0x0/0x66
Mar 13 06:30:35 node1 kernel: [902670.942115]  [<c1008d87>] ? kernel_thread_helper+0x7/0x10
Mar 13 06:30:35 node1 kernel: [902670.942117] Mem-Info:
Mar 13 06:30:35 node1 kernel: [902670.942118] DMA per-cpu:
Mar 13 06:30:35 node1 kernel: [902670.942119] CPU    0: hi:    0, btch:   1 usd:   0
Mar 13 06:30:35 node1 kernel: [902670.942121] CPU    1: hi:    0, btch:   1 usd:   0
Mar 13 06:30:35 node1 kernel: [902670.942122] Normal per-cpu:
Mar 13 06:30:35 node1 kernel: [902670.942124] CPU    0: hi:  186, btch:  31 usd: 180
Mar 13 06:30:35 node1 kernel: [902670.942125] CPU    1: hi:  186, btch:  31 usd: 154
Mar 13 06:30:35 node1 kernel: [902670.942126] HighMem per-cpu:
Mar 13 06:30:35 node1 kernel: [902670.942128] CPU    0: hi:  186, btch:  31 usd:  47
Mar 13 06:30:35 node1 kernel: [902670.942129] CPU    1: hi:  186, btch:  31 usd:  19
Mar 13 06:30:35 node1 kernel: [902670.942133] active_anon:137306 inactive_anon:39520 isolated_anon:0
Mar 13 06:30:35 node1 kernel: [902670.942133]  active_file:223625 inactive_file:375450 isolated_file:32
Mar 13 06:30:35 node1 kernel: [902670.942134]  unevictable:4083 dirty:1708 writeback:0 unstable:0
Mar 13 06:30:35 node1 kernel: [902670.942135]  free:187574 slab_reclaimable:50120 slab_unreclaimable:3993
Mar 13 06:30:35 node1 kernel: [902670.942136]  mapped:23637 shmem:11538 pagetables:2294 bounce:27
Mar 13 06:30:35 node1 kernel: [902670.942141] DMA free:3524kB min:64kB low:80kB high:96kB active_anon:0kB inactive_anon:0kB active_file:2728kB
inactive_file:6180kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15868kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB
slab_reclaimable:3512kB slab_unreclaimable:8kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0
all_unreclaimable? no
Mar 13 06:30:35 node1 kernel: [902670.942144] lowmem_reserve[]: 0 865 4030 4030
Mar 13 06:30:35 node1 kernel: [902670.942150] Normal free:40300kB min:3728kB low:4660kB high:5592kB active_anon:4280kB inactive_anon:15772kB
active_file:199036kB inactive_file:355444kB unevictable:0kB isolated(anon):0kB isolated(file):128kB present:885944kB mlocked:0kB dirty:228kB
writeback:0kB mapped:24kB shmem:8kB slab_reclaimable:196968kB slab_unreclaimable:15964kB kernel_stack:2608kB pagetables:104kB unstable:0kB
bounce:108kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Mar 13 06:30:35 node1 kernel: [902670.942155] lowmem_reserve[]: 0 0 25321 25321
Mar 13 06:30:35 node1 kernel: [902670.942160] HighMem free:706472kB min:512kB low:3920kB high:7332kB active_anon:544944kB inactive_anon:142308kB
active_file:692736kB inactive_file:1140176kB unevictable:16332kB isolated(anon):0kB isolated(file):0kB present:3241132kB mlocked:16332kB dirty:6604kB
writeback:0kB mapped:94524kB shmem:46144kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:9072kB unstable:0kB bounce:0kB
writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Mar 13 06:30:35 node1 kernel: [902670.942165] lowmem_reserve[]: 0 0 0 0
Mar 13 06:30:35 node1 kernel: [902670.942168] DMA: 15*4kB 15*8kB 9*16kB 22*32kB 7*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 1*2048kB 0*4096kB = 3524kB
Mar 13 06:30:35 node1 kernel: [902670.942174] Normal: 3507*4kB 3159*8kB 39*16kB 8*32kB 1*64kB 1*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 40372kB
Mar 13 06:30:35 node1 kernel: [902670.942180] HighMem: 78*4kB 56*8kB 23*16kB 12*32kB 2059*64kB 2550*128kB 704*256kB 104*512kB 13*1024kB 0*2048kB
0*4096kB = 706472kB
Mar 13 06:30:35 node1 kernel: [902670.942187] 612114 total pagecache pages
Mar 13 06:30:35 node1 kernel: [902670.942188] 160 pages in swap cache
Mar 13 06:30:35 node1 kernel: [902670.942190] Swap cache stats: add 21929, delete 21769, find 243894/245286
Mar 13 06:30:35 node1 kernel: [902670.942191] Free swap  = 4880236kB
Mar 13 06:30:35 node1 kernel: [902670.942192] Total swap = 4883640kB
Mar 13 06:30:35 node1 kernel: [902670.951693] 1245184 pages RAM
Mar 13 06:30:35 node1 kernel: [902670.951696] 1017858 pages HighMem
Mar 13 06:30:35 node1 kernel: [902670.951697] 210949 pages reserved
Mar 13 06:30:35 node1 kernel: [902670.951698] 327195 pages shared
Mar 13 06:30:35 node1 kernel: [902670.951699] 642296 pages non-shared
Mar 13 06:30:35 node1 kernel: [902670.951716] drbd1_receiver: page allocation failure. order:4, mode:0x4020
Mar 13 06:30:35 node1 kernel: [902670.951718] Pid: 25245, comm: drbd1_receiver Not tainted 2.6.32-5-vserver-686-bigmem #1
Mar 13 06:30:35 node1 kernel: [902670.951720] Call Trace:
Mar 13 06:30:35 node1 kernel: [902670.951725]  [<c109ad8c>] ? __alloc_pages_nodemask+0x484/0x4d9
Mar 13 06:30:35 node1 kernel: [902670.951728]  [<c109aded>] ? __get_free_pages+0xc/0x17
Mar 13 06:30:35 node1 kernel: [902670.951731]  [<c10bf261>] ? __kmalloc+0x30/0x128
Mar 13 06:30:35 node1 kernel: [902670.951740]  [<c11ee2f4>] ? pskb_expand_head+0x4f/0x157
Mar 13 06:30:35 node1 kernel: [902670.951742]  [<c11ee74b>] ? __pskb_pull_tail+0x40/0x200
Mar 13 06:30:35 node1 kernel: [902670.951744]  [<c11eb077>] ? sock_wfree+0x17/0x4b
Mar 13 06:30:35 node1 kernel: [902670.951747]  [<c11f555e>] ? dev_queue_xmit+0xe4/0x38e
Mar 13 06:30:35 node1 kernel: [902670.951750]  [<c1216cea>] ? ip_finish_output2+0x187/0x1c2
Mar 13 06:30:35 node1 kernel: [902670.951752]  [<c12161ed>] ? ip_local_out+0x15/0x17
Mar 13 06:30:35 node1 kernel: [902670.951754]  [<c12169cc>] ? ip_queue_xmit+0x31d/0x378
Mar 13 06:30:35 node1 kernel: [902670.951762]  [<f81b7696>] ? scsi_init_sgtable+0x2d/0x46 [scsi_mod]
Mar 13 06:30:35 node1 kernel: [902670.951765]  [<c1225c52>] ? tcp_transmit_skb+0x595/0x5cc
Mar 13 06:30:35 node1 kernel: [902670.951767]  [<c1227cb5>] ? tcp_write_xmit+0x7a3/0x874
Mar 13 06:30:35 node1 kernel: [902670.951769]  [<c12236e4>] ? tcp_ack+0x1611/0x1802
Mar 13 06:30:35 node1 kernel: [902670.951771]  [<c122448c>] ? tcp_rcv_established+0x5cf/0x62c
Mar 13 06:30:35 node1 kernel: [902670.951774]  [<c1224fdf>] ? tcp_established_options+0x1d/0x8b
Mar 13 06:30:35 node1 kernel: [902670.951776]  [<c12252a3>] ? tcp_current_mss+0x38/0x53
Mar 13 06:30:35 node1 kernel: [902670.951778]  [<c1227dcc>] ? __tcp_push_pending_frames+0x1e/0x50
Mar 13 06:30:35 node1 kernel: [902670.951780]  [<c12238f0>] ? tcp_data_snd_check+0x1b/0xd2
Mar 13 06:30:35 node1 kernel: [902670.951782]  [<c1224493>] ? tcp_rcv_established+0x5d6/0x62c
Mar 13 06:30:35 node1 kernel: [902670.951785]  [<c122a9ae>] ? tcp_v4_do_rcv+0x15f/0x2cf
Mar 13 06:30:35 node1 kernel: [902670.951787]  [<c11f2e36>] ? net_tx_action+0xa9/0xf9
Mar 13 06:30:35 node1 kernel: [902670.951789]  [<c11ea5ff>] ? release_sock+0x38/0x89
Mar 13 06:30:35 node1 kernel: [902670.951791]  [<c11eab8a>] ? sk_wait_data+0x55/0xa2
Mar 13 06:30:35 node1 kernel: [902670.951794]  [<c104b08a>] ? autoremove_wake_function+0x0/0x2d
Mar 13 06:30:35 node1 kernel: [902670.951797]  [<c121cf09>] ? tcp_recvmsg+0x3e4/0x8e5
Mar 13 06:30:35 node1 kernel: [902670.951799]  [<c11e9cdf>] ? sock_common_recvmsg+0x2f/0x45
Mar 13 06:30:35 node1 kernel: [902670.951801]  [<c11e9cdf>] ? sock_common_recvmsg+0x2f/0x45
Mar 13 06:30:35 node1 kernel: [902670.951805]  [<c11e7fc3>] ? __sock_recvmsg+0x50/0xbb
Mar 13 06:30:35 node1 kernel: [902670.951807]  [<c11e855d>] ? sock_recvmsg+0x9d/0xb6
Mar 13 06:30:35 node1 kernel: [902670.951809]  [<c104b08a>] ? autoremove_wake_function+0x0/0x2d
Mar 13 06:30:35 node1 kernel: [902670.951812]  [<c113c658>] ? __make_request+0x364/0x3d9
Mar 13 06:30:35 node1 kernel: [902670.951818]  [<f9067ddf>] ? drbd_recv+0x62/0x106 [drbd]
Mar 13 06:30:35 node1 kernel: [902670.951822]  [<f9067e95>] ? drbd_recv_header+0x12/0xa5 [drbd]
Mar 13 06:30:35 node1 kernel: [902670.951827]  [<f906e669>] ? drbdd_init+0xf3/0x245 [drbd]
Mar 13 06:30:35 node1 kernel: [902670.951831]  [<f9079bf7>] ? drbd_thread_setup+0x1f/0xcc [drbd]
Mar 13 06:30:35 node1 kernel: [902670.951836]  [<f9079bd8>] ? drbd_thread_setup+0x0/0xcc [drbd]
Mar 13 06:30:35 node1 kernel: [902670.951838]  [<c104ae58>] ? kthread+0x61/0x66
Mar 13 06:30:35 node1 kernel: [902670.951840]  [<c104adf7>] ? kthread+0x0/0x66
Mar 13 06:30:35 node1 kernel: [902670.951843]  [<c1008d87>] ? kernel_thread_helper+0x7/0x10
Mar 13 06:30:35 node1 kernel: [902670.951844] Mem-Info:
Mar 13 06:30:35 node1 kernel: [902670.951845] DMA per-cpu:
Mar 13 06:30:35 node1 kernel: [902670.951847] CPU    0: hi:    0, btch:   1 usd:   0
Mar 13 06:30:35 node1 kernel: [902670.951848] CPU    1: hi:    0, btch:   1 usd:   0
Mar 13 06:30:35 node1 kernel: [902670.951850] Normal per-cpu:
Mar 13 06:30:35 node1 kernel: [902670.951851] CPU    0: hi:  186, btch:  31 usd: 180
Mar 13 06:30:35 node1 kernel: [902670.951853] CPU    1: hi:  186, btch:  31 usd: 138
Mar 13 06:30:35 node1 kernel: [902670.951855] HighMem per-cpu:
Mar 13 06:30:35 node1 kernel: [902670.951856] CPU    0: hi:  186, btch:  31 usd:  47
Mar 13 06:30:35 node1 kernel: [902670.951858] CPU    1: hi:  186, btch:  31 usd:  10
Mar 13 06:30:35 node1 kernel: [902670.951861] active_anon:137306 inactive_anon:39520 isolated_anon:0
Mar 13 06:30:35 node1 kernel: [902670.951862]  active_file:223625 inactive_file:375487 isolated_file:32
Mar 13 06:30:35 node1 kernel: [902670.951863]  unevictable:4083 dirty:1745 writeback:0 unstable:0
Mar 13 06:30:35 node1 kernel: [902670.951863]  free:187543 slab_reclaimable:50120 slab_unreclaimable:3993
Mar 13 06:30:35 node1 kernel: [902670.951864]  mapped:23637 shmem:11538 pagetables:2294 bounce:52
Mar 13 06:30:35 node1 kernel: [902670.951869] DMA free:3524kB min:64kB low:80kB high:96kB active_anon:0kB inactive_anon:0kB active_file:2728kB
inactive_file:6180kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15868kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB
slab_reclaimable:3512kB slab_unreclaimable:8kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0
all_unreclaimable? no
Mar 13 06:30:35 node1 kernel: [902670.951873] lowmem_reserve[]: 0 865 4030 4030
Mar 13 06:30:35 node1 kernel: [902670.951879] Normal free:40300kB min:3728kB low:4660kB high:5592kB active_anon:4280kB inactive_anon:15772kB
active_file:199036kB inactive_file:355444kB unevictable:0kB isolated(anon):0kB isolated(file):128kB present:885944kB mlocked:0kB dirty:228kB
writeback:0kB mapped:24kB shmem:8kB slab_reclaimable:196968kB slab_unreclaimable:15964kB kernel_stack:2608kB pagetables:104kB unstable:0kB
bounce:208kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Mar 13 06:30:35 node1 kernel: [902670.951883] lowmem_reserve[]: 0 0 25321 25321
Mar 13 06:30:35 node1 kernel: [902670.951889] HighMem free:706348kB min:512kB low:3920kB high:7332kB active_anon:544944kB inactive_anon:142308kB
active_file:692736kB inactive_file:1140324kB unevictable:16332kB isolated(anon):0kB isolated(file):0kB present:3241132kB mlocked:16332kB dirty:6752kB
writeback:0kB mapped:94524kB shmem:46144kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:9072kB unstable:0kB bounce:0kB
writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Mar 13 06:30:35 node1 kernel: [902670.951893] lowmem_reserve[]: 0 0 0 0
Mar 13 06:30:35 node1 kernel: [902670.951896] DMA: 15*4kB 15*8kB 9*16kB 22*32kB 7*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 1*2048kB 0*4096kB = 3524kB
Mar 13 06:30:35 node1 kernel: [902670.951902] Normal: 3507*4kB 3158*8kB 39*16kB 8*32kB 1*64kB 1*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 40364kB
Mar 13 06:30:35 node1 kernel: [902670.951908] HighMem: 77*4kB 57*8kB 23*16kB 12*32kB 2057*64kB 2550*128kB 704*256kB 104*512kB 13*1024kB 0*2048kB
0*4096kB = 706348kB
Mar 13 06:30:35 node1 kernel: [902670.951915] 612151 total pagecache pages
Mar 13 06:30:35 node1 kernel: [902670.951917] 160 pages in swap cache
Mar 13 06:30:35 node1 kernel: [902670.951918] Swap cache stats: add 21929, delete 21769, find 243894/245286
Mar 13 06:30:35 node1 kernel: [902670.951920] Free swap  = 4880236kB
Mar 13 06:30:35 node1 kernel: [902670.951921] Total swap = 4883640kB
Mar 13 06:30:35 node1 kernel: [902670.961279] 1245184 pages RAM
Mar 13 06:30:35 node1 kernel: [902670.961280] 1017858 pages HighMem
Mar 13 06:30:35 node1 kernel: [902670.961281] 210949 pages reserved
Mar 13 06:30:35 node1 kernel: [902670.961282] 327205 pages shared
Mar 13 06:30:35 node1 kernel: [902670.961283] 642316 pages non-shared
Mar 13 06:30:35 node1 kernel: [902670.961299] drbd1_receiver: page allocation failure. order:4, mode:0x4020
Mar 13 06:30:35 node1 kernel: [902670.961302] Pid: 25245, comm: drbd1_receiver Not tainted 2.6.32-5-vserver-686-bigmem #1
Mar 13 06:30:35 node1 kernel: [902670.961304] Call Trace:
Mar 13 06:30:35 node1 kernel: [902670.961309]  [<c109ad8c>] ? __alloc_pages_nodemask+0x484/0x4d9
Mar 13 06:30:35 node1 kernel: [902670.961312]  [<c109aded>] ? __get_free_pages+0xc/0x17
Mar 13 06:30:35 node1 kernel: [902670.961315]  [<c10bf261>] ? __kmalloc+0x30/0x128
Mar 13 06:30:35 node1 kernel: [902670.961324]  [<c11ee2f4>] ? pskb_expand_head+0x4f/0x157
Mar 13 06:30:35 node1 kernel: [902670.961327]  [<c11ee74b>] ? __pskb_pull_tail+0x40/0x200
Mar 13 06:30:35 node1 kernel: [902670.961329]  [<c11eb077>] ? sock_wfree+0x17/0x4b
Mar 13 06:30:35 node1 kernel: [902670.961332]  [<c11f555e>] ? dev_queue_xmit+0xe4/0x38e
Mar 13 06:30:35 node1 kernel: [902670.961334]  [<c1216cea>] ? ip_finish_output2+0x187/0x1c2
Mar 13 06:30:35 node1 kernel: [902670.961337]  [<c12161ed>] ? ip_local_out+0x15/0x17
Mar 13 06:30:35 node1 kernel: [902670.961339]  [<c12169cc>] ? ip_queue_xmit+0x31d/0x378
Mar 13 06:30:35 node1 kernel: [902670.961346]  [<f81b7696>] ? scsi_init_sgtable+0x2d/0x46 [scsi_mod]
Mar 13 06:30:35 node1 kernel: [902670.961349]  [<c1225c52>] ? tcp_transmit_skb+0x595/0x5cc
Mar 13 06:30:35 node1 kernel: [902670.961351]  [<c1227cb5>] ? tcp_write_xmit+0x7a3/0x874
Mar 13 06:30:35 node1 kernel: [902670.961353]  [<c12236e4>] ? tcp_ack+0x1611/0x1802
Mar 13 06:30:35 node1 kernel: [902670.961356]  [<c1224fdf>] ? tcp_established_options+0x1d/0x8b
Mar 13 06:30:35 node1 kernel: [902670.961358]  [<c12252a3>] ? tcp_current_mss+0x38/0x53
Mar 13 06:30:35 node1 kernel: [902670.961360]  [<c1227dcc>] ? __tcp_push_pending_frames+0x1e/0x50
Mar 13 06:30:35 node1 kernel: [902670.961362]  [<c12238f0>] ? tcp_data_snd_check+0x1b/0xd2
Mar 13 06:30:35 node1 kernel: [902670.961364]  [<c1224493>] ? tcp_rcv_established+0x5d6/0x62c
Mar 13 06:30:35 node1 kernel: [902670.961367]  [<c122a9ae>] ? tcp_v4_do_rcv+0x15f/0x2cf
Mar 13 06:30:35 node1 kernel: [902670.961369]  [<c11f2e36>] ? net_tx_action+0xa9/0xf9
Mar 13 06:30:35 node1 kernel: [902670.961371]  [<c11ea5ff>] ? release_sock+0x38/0x89
Mar 13 06:30:35 node1 kernel: [902670.961373]  [<c11eab8a>] ? sk_wait_data+0x55/0xa2
Mar 13 06:30:35 node1 kernel: [902670.961376]  [<c104b08a>] ? autoremove_wake_function+0x0/0x2d
Mar 13 06:30:35 node1 kernel: [902670.961379]  [<c121cf09>] ? tcp_recvmsg+0x3e4/0x8e5
Mar 13 06:30:35 node1 kernel: [902670.961381]  [<c11e9cdf>] ? sock_common_recvmsg+0x2f/0x45
Mar 13 06:30:35 node1 kernel: [902670.961383]  [<c11e9cdf>] ? sock_common_recvmsg+0x2f/0x45
Mar 13 06:30:35 node1 kernel: [902670.961387]  [<c11e7fc3>] ? __sock_recvmsg+0x50/0xbb
Mar 13 06:30:35 node1 kernel: [902670.961389]  [<c11e855d>] ? sock_recvmsg+0x9d/0xb6
Mar 13 06:30:35 node1 kernel: [902670.961397]  [<c104b08a>] ? autoremove_wake_function+0x0/0x2d
Mar 13 06:30:35 node1 kernel: [902670.961400]  [<c113c658>] ? __make_request+0x364/0x3d9
Mar 13 06:30:35 node1 kernel: [902670.961406]  [<f9067ddf>] ? drbd_recv+0x62/0x106 [drbd]
Mar 13 06:30:35 node1 kernel: [902670.961410]  [<f9067e95>] ? drbd_recv_header+0x12/0xa5 [drbd]
Mar 13 06:30:35 node1 kernel: [902670.961415]  [<f906e669>] ? drbdd_init+0xf3/0x245 [drbd]
Mar 13 06:30:35 node1 kernel: [902670.961419]  [<f9079bf7>] ? drbd_thread_setup+0x1f/0xcc [drbd]
Mar 13 06:30:35 node1 kernel: [902670.961424]  [<f9079bd8>] ? drbd_thread_setup+0x0/0xcc [drbd]
Mar 13 06:30:35 node1 kernel: [902670.961426]  [<c104ae58>] ? kthread+0x61/0x66
Mar 13 06:30:35 node1 kernel: [902670.961428]  [<c104adf7>] ? kthread+0x0/0x66
Mar 13 06:30:35 node1 kernel: [902670.961431]  [<c1008d87>] ? kernel_thread_helper+0x7/0x10
Mar 13 06:30:35 node1 kernel: [902670.961432] Mem-Info:
Mar 13 06:30:35 node1 kernel: [902670.961433] DMA per-cpu:
Mar 13 06:30:35 node1 kernel: [902670.961435] CPU    0: hi:    0, btch:   1 usd:   0
Mar 13 06:30:35 node1 kernel: [902670.961436] CPU    1: hi:    0, btch:   1 usd:   0
Mar 13 06:30:35 node1 kernel: [902670.961438] Normal per-cpu:
Mar 13 06:30:35 node1 kernel: [902670.961439] CPU    0: hi:  186, btch:  31 usd: 180
Mar 13 06:30:35 node1 kernel: [902670.961441] CPU    1: hi:  186, btch:  31 usd: 136
Mar 13 06:30:35 node1 kernel: [902670.961442] HighMem per-cpu:
Mar 13 06:30:35 node1 kernel: [902670.961443] CPU    0: hi:  186, btch:  31 usd:  47
Mar 13 06:30:35 node1 kernel: [902670.961445] CPU    1: hi:  186, btch:  31 usd:   2
Mar 13 06:30:35 node1 kernel: [902670.961448] active_anon:137306 inactive_anon:39520 isolated_anon:0
Mar 13 06:30:35 node1 kernel: [902670.961449]  active_file:223625 inactive_file:375487 isolated_file:32
Mar 13 06:30:35 node1 kernel: [902670.961450]  unevictable:4083 dirty:1745 writeback:0 unstable:0
Mar 13 06:30:35 node1 kernel: [902670.961451]  free:187543 slab_reclaimable:50120 slab_unreclaimable:3993
Mar 13 06:30:35 node1 kernel: [902670.961451]  mapped:23637 shmem:11538 pagetables:2294 bounce:52
Mar 13 06:30:35 node1 kernel: [902670.961456] DMA free:3524kB min:64kB low:80kB high:96kB active_anon:0kB inactive_anon:0kB active_file:2728kB
inactive_file:6180kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15868kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB
slab_reclaimable:3512kB slab_unreclaimable:8kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0
all_unreclaimable? no
Mar 13 06:30:35 node1 kernel: [902670.961460] lowmem_reserve[]: 0 865 4030 4030
Mar 13 06:30:35 node1 kernel: [902670.961466] Normal free:40300kB min:3728kB low:4660kB high:5592kB active_anon:4280kB inactive_anon:15772kB
active_file:199036kB inactive_file:355444kB unevictable:0kB isolated(anon):0kB isolated(file):128kB present:885944kB mlocked:0kB dirty:228kB
writeback:0kB mapped:24kB shmem:8kB slab_reclaimable:196968kB slab_unreclaimable:15964kB kernel_stack:2608kB pagetables:104kB unstable:0kB
bounce:208kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Mar 13 06:30:35 node1 kernel: [902670.961470] lowmem_reserve[]: 0 0 25321 25321
Mar 13 06:30:35 node1 kernel: [902670.961476] HighMem free:706348kB min:512kB low:3920kB high:7332kB active_anon:544944kB inactive_anon:142308kB
active_file:692736kB inactive_file:1140324kB unevictable:16332kB isolated(anon):0kB isolated(file):0kB present:3241132kB mlocked:16332kB dirty:6752kB
writeback:0kB mapped:94524kB shmem:46144kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:9072kB unstable:0kB bounce:0kB
writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Mar 13 06:30:35 node1 kernel: [902670.961480] lowmem_reserve[]: 0 0 0 0
Mar 13 06:30:35 node1 kernel: [902670.961483] DMA: 15*4kB 15*8kB 9*16kB 22*32kB 7*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 1*2048kB 0*4096kB = 3524kB
Mar 13 06:30:35 node1 kernel: [902670.961489] Normal: 3507*4kB 3158*8kB 39*16kB 8*32kB 1*64kB 1*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 40364kB
Mar 13 06:30:35 node1 kernel: [902670.961495] HighMem: 77*4kB 57*8kB 23*16kB 12*32kB 2057*64kB 2550*128kB 704*256kB 104*512kB 13*1024kB 0*2048kB
0*4096kB = 706348kB
Mar 13 06:30:35 node1 kernel: [902670.961502] 612188 total pagecache pages
Mar 13 06:30:35 node1 kernel: [902670.961503] 160 pages in swap cache
Mar 13 06:30:35 node1 kernel: [902670.961505] Swap cache stats: add 21929, delete 21769, find 243894/245286
Mar 13 06:30:35 node1 kernel: [902670.961506] Free swap  = 4880236kB
Mar 13 06:30:35 node1 kernel: [902670.961507] Total swap = 4883640kB
Mar 13 06:30:35 node1 kernel: [902670.970841] 1245184 pages RAM
Mar 13 06:30:35 node1 kernel: [902670.970842] 1017858 pages HighMem
Mar 13 06:30:35 node1 kernel: [902670.970843] 210949 pages reserved
Mar 13 06:30:35 node1 kernel: [902670.970844] 327209 pages shared
Mar 13 06:30:35 node1 kernel: [902670.970845] 642318 pages non-shared
Mar 13 06:30:35 node1 kernel: [902670.970862] drbd1_receiver: page allocation failure. order:4, mode:0x4020
Mar 13 06:30:35 node1 kernel: [902670.970865] Pid: 25245, comm: drbd1_receiver Not tainted 2.6.32-5-vserver-686-bigmem #1
Mar 13 06:30:35 node1 kernel: [902670.970866] Call Trace:
Mar 13 06:30:35 node1 kernel: [902670.970872]  [<c109ad8c>] ? __alloc_pages_nodemask+0x484/0x4d9
Mar 13 06:30:35 node1 kernel: [902670.970875]  [<c109aded>] ? __get_free_pages+0xc/0x17
Mar 13 06:30:35 node1 kernel: [902670.970877]  [<c10bf261>] ? __kmalloc+0x30/0x128
Mar 13 06:30:35 node1 kernel: [902670.970886]  [<c11ee2f4>] ? pskb_expand_head+0x4f/0x157
Mar 13 06:30:35 node1 kernel: [902670.970889]  [<c11ee74b>] ? __pskb_pull_tail+0x40/0x200
Mar 13 06:30:35 node1 kernel: [902670.970891]  [<c11eb077>] ? sock_wfree+0x17/0x4b
Mar 13 06:30:35 node1 kernel: [902670.970893]  [<c11f555e>] ? dev_queue_xmit+0xe4/0x38e
Mar 13 06:30:35 node1 kernel: [902670.970896]  [<c1216cea>] ? ip_finish_output2+0x187/0x1c2
Mar 13 06:30:35 node1 kernel: [902670.970898]  [<c12161ed>] ? ip_local_out+0x15/0x17
Mar 13 06:30:35 node1 kernel: [902670.970901]  [<c12169cc>] ? ip_queue_xmit+0x31d/0x378
Mar 13 06:30:35 node1 kernel: [902670.970908]  [<f81b7696>] ? scsi_init_sgtable+0x2d/0x46 [scsi_mod]
Mar 13 06:30:35 node1 kernel: [902670.970911]  [<c1225c52>] ? tcp_transmit_skb+0x595/0x5cc
Mar 13 06:30:35 node1 kernel: [902670.970913]  [<c1227cb5>] ? tcp_write_xmit+0x7a3/0x874
Mar 13 06:30:35 node1 kernel: [902670.970915]  [<c12236e4>] ? tcp_ack+0x1611/0x1802
Mar 13 06:30:35 node1 kernel: [902670.970918]  [<c122448c>] ? tcp_rcv_established+0x5cf/0x62c
Mar 13 06:30:35 node1 kernel: [902670.970920]  [<c1224fdf>] ? tcp_established_options+0x1d/0x8b
Mar 13 06:30:35 node1 kernel: [902670.970922]  [<c12252a3>] ? tcp_current_mss+0x38/0x53
Mar 13 06:30:35 node1 kernel: [902670.970924]  [<c1227dcc>] ? __tcp_push_pending_frames+0x1e/0x50
Mar 13 06:30:35 node1 kernel: [902670.970926]  [<c12238f0>] ? tcp_data_snd_check+0x1b/0xd2
Mar 13 06:30:35 node1 kernel: [902670.970928]  [<c1224493>] ? tcp_rcv_established+0x5d6/0x62c
Mar 13 06:30:35 node1 kernel: [902670.970931]  [<c122a9ae>] ? tcp_v4_do_rcv+0x15f/0x2cf
Mar 13 06:30:35 node1 kernel: [902670.970933]  [<c11f2e36>] ? net_tx_action+0xa9/0xf9
Mar 13 06:30:35 node1 kernel: [902670.970936]  [<c11ea5ff>] ? release_sock+0x38/0x89
Mar 13 06:30:35 node1 kernel: [902670.970938]  [<c11eab8a>] ? sk_wait_data+0x55/0xa2
Mar 13 06:30:35 node1 kernel: [902670.970941]  [<c104b08a>] ? autoremove_wake_function+0x0/0x2d
Mar 13 06:30:35 node1 kernel: [902670.970943]  [<c121cf09>] ? tcp_recvmsg+0x3e4/0x8e5
Mar 13 06:30:35 node1 kernel: [902670.970946]  [<c11e9cdf>] ? sock_common_recvmsg+0x2f/0x45
Mar 13 06:30:35 node1 kernel: [902670.970948]  [<c11e9cdf>] ? sock_common_recvmsg+0x2f/0x45
Mar 13 06:30:35 node1 kernel: [902670.970952]  [<c11e7fc3>] ? __sock_recvmsg+0x50/0xbb
Mar 13 06:30:35 node1 kernel: [902670.970954]  [<c11e855d>] ? sock_recvmsg+0x9d/0xb6
Mar 13 06:30:35 node1 kernel: [902670.970956]  [<c104b08a>] ? autoremove_wake_function+0x0/0x2d
Mar 13 06:30:35 node1 kernel: [902670.970959]  [<c113c658>] ? __make_request+0x364/0x3d9
Mar 13 06:30:35 node1 kernel: [902670.970965]  [<f9067ddf>] ? drbd_recv+0x62/0x106 [drbd]
Mar 13 06:30:35 node1 kernel: [902670.970969]  [<f9067e95>] ? drbd_recv_header+0x12/0xa5 [drbd]
Mar 13 06:30:35 node1 kernel: [902670.970974]  [<f906e669>] ? drbdd_init+0xf3/0x245 [drbd]
Mar 13 06:30:35 node1 kernel: [902670.970979]  [<f9079bf7>] ? drbd_thread_setup+0x1f/0xcc [drbd]
Mar 13 06:30:35 node1 kernel: [902670.970983]  [<f9079bd8>] ? drbd_thread_setup+0x0/0xcc [drbd]
Mar 13 06:30:35 node1 kernel: [902670.970985]  [<c104ae58>] ? kthread+0x61/0x66
Mar 13 06:30:35 node1 kernel: [902670.970987]  [<c104adf7>] ? kthread+0x0/0x66
Mar 13 06:30:35 node1 kernel: [902670.970990]  [<c1008d87>] ? kernel_thread_helper+0x7/0x10
Mar 13 06:30:35 node1 kernel: [902670.970992] Mem-Info:
Mar 13 06:30:35 node1 kernel: [902670.970993] DMA per-cpu:
Mar 13 06:30:35 node1 kernel: [902670.970994] CPU    0: hi:    0, btch:   1 usd:   0
Mar 13 06:30:35 node1 kernel: [902670.970996] CPU    1: hi:    0, btch:   1 usd:   0
Mar 13 06:30:35 node1 kernel: [902670.970997] Normal per-cpu:
Mar 13 06:30:35 node1 kernel: [902670.970998] CPU    0: hi:  186, btch:  31 usd: 180
Mar 13 06:30:35 node1 kernel: [902670.971000] CPU    1: hi:  186, btch:  31 usd: 135
Mar 13 06:30:35 node1 kernel: [902670.971001] HighMem per-cpu:
Mar 13 06:30:35 node1 kernel: [902670.971002] CPU    0: hi:  186, btch:  31 usd:  47
Mar 13 06:30:35 node1 kernel: [902670.971004] CPU    1: hi:  186, btch:  31 usd:  56
Mar 13 06:30:35 node1 kernel: [902670.971007] active_anon:137306 inactive_anon:39520 isolated_anon:0
Mar 13 06:30:35 node1 kernel: [902670.971008]  active_file:223625 inactive_file:375487 isolated_file:32
Mar 13 06:30:35 node1 kernel: [902670.971009]  unevictable:4083 dirty:1745 writeback:0 unstable:0
Mar 13 06:30:35 node1 kernel: [902670.971010]  free:187481 slab_reclaimable:50120 slab_unreclaimable:3993
Mar 13 06:30:35 node1 kernel: [902670.971011]  mapped:23637 shmem:11538 pagetables:2294 bounce:52
Mar 13 06:30:35 node1 kernel: [902670.971015] DMA free:3524kB min:64kB low:80kB high:96kB active_anon:0kB inactive_anon:0kB active_file:2728kB
inactive_file:6180kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15868kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB
slab_reclaimable:3512kB slab_unreclaimable:8kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0
all_unreclaimable? no
Mar 13 06:30:35 node1 kernel: [902670.971019] lowmem_reserve[]: 0 865 4030 4030
Mar 13 06:30:35 node1 kernel: [902670.971025] Normal free:40300kB min:3728kB low:4660kB high:5592kB active_anon:4280kB inactive_anon:15772kB
active_file:199036kB inactive_file:355444kB unevictable:0kB isolated(anon):0kB isolated(file):128kB present:885944kB mlocked:0kB dirty:228kB
writeback:0kB mapped:24kB shmem:8kB slab_reclaimable:196968kB slab_unreclaimable:15964kB kernel_stack:2608kB pagetables:104kB unstable:0kB
bounce:208kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Mar 13 06:30:35 node1 kernel: [902670.971029] lowmem_reserve[]: 0 0 25321 25321
Mar 13 06:30:35 node1 kernel: [902670.971035] HighMem free:706100kB min:512kB low:3920kB high:7332kB active_anon:544944kB inactive_anon:142308kB
active_file:692736kB inactive_file:1140324kB unevictable:16332kB isolated(anon):0kB isolated(file):0kB present:3241132kB mlocked:16332kB dirty:6752kB
writeback:0kB mapped:94524kB shmem:46144kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:9072kB unstable:0kB bounce:0kB
writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Mar 13 06:30:35 node1 kernel: [902670.971039] lowmem_reserve[]: 0 0 0 0
Mar 13 06:30:35 node1 kernel: [902670.971042] DMA: 15*4kB 15*8kB 9*16kB 22*32kB 7*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 1*2048kB 0*4096kB = 3524kB
Mar 13 06:30:35 node1 kernel: [902670.971048] Normal: 3507*4kB 3158*8kB 39*16kB 8*32kB 1*64kB 1*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 40364kB
Mar 13 06:30:35 node1 kernel: [902670.971054] HighMem: 47*4kB 57*8kB 23*16kB 12*32kB 2055*64kB 2550*128kB 704*256kB 104*512kB 13*1024kB 0*2048kB
0*4096kB = 706100kB
Mar 13 06:30:35 node1 kernel: [902670.971061] 612188 total pagecache pages
Mar 13 06:30:35 node1 kernel: [902670.971062] 160 pages in swap cache
Mar 13 06:30:35 node1 kernel: [902670.971064] Swap cache stats: add 21929, delete 21769, find 243894/245286
Mar 13 06:30:35 node1 kernel: [902670.971065] Free swap  = 4880236kB
Mar 13 06:30:35 node1 kernel: [902670.971066] Total swap = 4883640kB
Mar 13 06:30:36 node1 kernel: [902670.980377] 1245184 pages RAM
Mar 13 06:30:36 node1 kernel: [902670.980378] 1017858 pages HighMem
Mar 13 06:30:36 node1 kernel: [902670.980379] 210949 pages reserved
Mar 13 06:30:36 node1 kernel: [902670.980380] 327217 pages shared
Mar 13 06:30:36 node1 kernel: [902670.980381] 642319 pages non-shared
Mar 13 06:30:36 node1 kernel: [902670.980397] drbd1_receiver: page allocation failure. order:4, mode:0x4020
Mar 13 06:30:36 node1 kernel: [902670.980399] Pid: 25245, comm: drbd1_receiver Not tainted 2.6.32-5-vserver-686-bigmem #1
Mar 13 06:30:36 node1 kernel: [902670.980401] Call Trace:
Mar 13 06:30:36 node1 kernel: [902670.980406]  [<c109ad8c>] ? __alloc_pages_nodemask+0x484/0x4d9
Mar 13 06:30:36 node1 kernel: [902670.980409]  [<c109aded>] ? __get_free_pages+0xc/0x17
Mar 13 06:30:36 node1 kernel: [902670.980412]  [<c10bf261>] ? __kmalloc+0x30/0x128
Mar 13 06:30:36 node1 kernel: [902670.980422]  [<c11ee2f4>] ? pskb_expand_head+0x4f/0x157
Mar 13 06:30:36 node1 kernel: [902670.980424]  [<c11ee74b>] ? __pskb_pull_tail+0x40/0x200
Mar 13 06:30:36 node1 kernel: [902670.980427]  [<c11eb077>] ? sock_wfree+0x17/0x4b
Mar 13 06:30:36 node1 kernel: [902670.980429]  [<c11f555e>] ? dev_queue_xmit+0xe4/0x38e
Mar 13 06:30:36 node1 kernel: [902670.980432]  [<c1216cea>] ? ip_finish_output2+0x187/0x1c2
Mar 13 06:30:36 node1 kernel: [902670.980434]  [<c12161ed>] ? ip_local_out+0x15/0x17
Mar 13 06:30:36 node1 kernel: [902670.980436]  [<c12169cc>] ? ip_queue_xmit+0x31d/0x378
Mar 13 06:30:36 node1 kernel: [902670.980443]  [<f81b7696>] ? scsi_init_sgtable+0x2d/0x46 [scsi_mod]
Mar 13 06:30:36 node1 kernel: [902670.980447]  [<c1225c52>] ? tcp_transmit_skb+0x595/0x5cc
Mar 13 06:30:36 node1 kernel: [902670.980449]  [<c1227cb5>] ? tcp_write_xmit+0x7a3/0x874
Mar 13 06:30:36 node1 kernel: [902670.980451]  [<c12236e4>] ? tcp_ack+0x1611/0x1802
Mar 13 06:30:36 node1 kernel: [902670.980453]  [<c1224fdf>] ? tcp_established_options+0x1d/0x8b
Mar 13 06:30:36 node1 kernel: [902670.980456]  [<c12252a3>] ? tcp_current_mss+0x38/0x53
Mar 13 06:30:36 node1 kernel: [902670.980458]  [<c1227dcc>] ? __tcp_push_pending_frames+0x1e/0x50
Mar 13 06:30:36 node1 kernel: [902670.980460]  [<c12238f0>] ? tcp_data_snd_check+0x1b/0xd2
Mar 13 06:30:36 node1 kernel: [902670.980462]  [<c1224493>] ? tcp_rcv_established+0x5d6/0x62c
Mar 13 06:30:36 node1 kernel: [902670.980465]  [<c122a9ae>] ? tcp_v4_do_rcv+0x15f/0x2cf
Mar 13 06:30:36 node1 kernel: [902670.980467]  [<c11f2e36>] ? net_tx_action+0xa9/0xf9
Mar 13 06:30:36 node1 kernel: [902670.980469]  [<c11ea5ff>] ? release_sock+0x38/0x89
Mar 13 06:30:36 node1 kernel: [902670.980471]  [<c11eab8a>] ? sk_wait_data+0x55/0xa2
Mar 13 06:30:36 node1 kernel: [902670.980474]  [<c104b08a>] ? autoremove_wake_function+0x0/0x2d
Mar 13 06:30:36 node1 kernel: [902670.980477]  [<c121cf09>] ? tcp_recvmsg+0x3e4/0x8e5
Mar 13 06:30:36 node1 kernel: [902670.980479]  [<c11e9cdf>] ? sock_common_recvmsg+0x2f/0x45
Mar 13 06:30:36 node1 kernel: [902670.980481]  [<c11e9cdf>] ? sock_common_recvmsg+0x2f/0x45
Mar 13 06:30:36 node1 kernel: [902670.980486]  [<c11e7fc3>] ? __sock_recvmsg+0x50/0xbb
Mar 13 06:30:36 node1 kernel: [902670.980488]  [<c11e855d>] ? sock_recvmsg+0x9d/0xb6
Mar 13 06:30:36 node1 kernel: [902670.980490]  [<c104b08a>] ? autoremove_wake_function+0x0/0x2d
Mar 13 06:30:36 node1 kernel: [902670.980494]  [<c113c658>] ? __make_request+0x364/0x3d9
Mar 13 06:30:36 node1 kernel: [902670.980499]  [<f9067ddf>] ? drbd_recv+0x62/0x106 [drbd]
Mar 13 06:30:36 node1 kernel: [902670.980504]  [<f9067e95>] ? drbd_recv_header+0x12/0xa5 [drbd]
Mar 13 06:30:36 node1 kernel: [902670.980508]  [<f906e669>] ? drbdd_init+0xf3/0x245 [drbd]
Mar 13 06:30:36 node1 kernel: [902670.980513]  [<f9079bf7>] ? drbd_thread_setup+0x1f/0xcc [drbd]
Mar 13 06:30:36 node1 kernel: [902670.980517]  [<f9079bd8>] ? drbd_thread_setup+0x0/0xcc [drbd]
Mar 13 06:30:36 node1 kernel: [902670.980519]  [<c104ae58>] ? kthread+0x61/0x66
Mar 13 06:30:36 node1 kernel: [902670.980522]  [<c104adf7>] ? kthread+0x0/0x66
Mar 13 06:30:36 node1 kernel: [902670.980525]  [<c1008d87>] ? kernel_thread_helper+0x7/0x10
Mar 13 06:30:36 node1 kernel: [902670.980526] Mem-Info:
Mar 13 06:30:36 node1 kernel: [902670.980527] DMA per-cpu:
Mar 13 06:30:36 node1 kernel: [902670.980529] CPU    0: hi:    0, btch:   1 usd:   0
Mar 13 06:30:36 node1 kernel: [902670.980530] CPU    1: hi:    0, btch:   1 usd:   0
Mar 13 06:30:36 node1 kernel: [902670.980531] Normal per-cpu:
Mar 13 06:30:36 node1 kernel: [902670.980533] CPU    0: hi:  186, btch:  31 usd: 180
Mar 13 06:30:36 node1 kernel: [902670.980534] CPU    1: hi:  186, btch:  31 usd: 135
Mar 13 06:30:36 node1 kernel: [902670.980535] HighMem per-cpu:
Mar 13 06:30:36 node1 kernel: [902670.980537] CPU    0: hi:  186, btch:  31 usd:  47
Mar 13 06:30:36 node1 kernel: [902670.980538] CPU    1: hi:  186, btch:  31 usd:  50
Mar 13 06:30:36 node1 kernel: [902670.980542] active_anon:137306 inactive_anon:39520 isolated_anon:0
Mar 13 06:30:36 node1 kernel: [902670.980542]  active_file:223625 inactive_file:375487 isolated_file:32
Mar 13 06:30:36 node1 kernel: [902670.980543]  unevictable:4083 dirty:1745 writeback:0 unstable:0
Mar 13 06:30:36 node1 kernel: [902670.980544]  free:187481 slab_reclaimable:50120 slab_unreclaimable:3993
Mar 13 06:30:36 node1 kernel: [902670.980545]  mapped:23637 shmem:11538 pagetables:2294 bounce:52
Mar 13 06:30:36 node1 kernel: [902670.980550] DMA free:3524kB min:64kB low:80kB high:96kB active_anon:0kB inactive_anon:0kB active_file:2728kB
inactive_file:6180kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15868kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB
slab_reclaimable:3512kB slab_unreclaimable:8kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0
all_unreclaimable? no
Mar 13 06:30:36 node1 kernel: [902670.980554] lowmem_reserve[]: 0 865 4030 4030
Mar 13 06:30:36 node1 kernel: [902670.980560] Normal free:40300kB min:3728kB low:4660kB high:5592kB active_anon:4280kB inactive_anon:15772kB
active_file:199036kB inactive_file:355444kB unevictable:0kB isolated(anon):0kB isolated(file):128kB present:885944kB mlocked:0kB dirty:228kB
writeback:0kB mapped:24kB shmem:8kB slab_reclaimable:196968kB slab_unreclaimable:15964kB kernel_stack:2608kB pagetables:104kB unstable:0kB
bounce:208kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Mar 13 06:30:36 node1 kernel: [902670.980564] lowmem_reserve[]: 0 0 25321 25321
Mar 13 06:30:36 node1 kernel: [902670.980569] HighMem free:706100kB min:512kB low:3920kB high:7332kB active_anon:544944kB inactive_anon:142308kB
active_file:692736kB inactive_file:1140324kB unevictable:16332kB isolated(anon):0kB isolated(file):0kB present:3241132kB mlocked:16332kB dirty:6752kB
writeback:0kB mapped:94524kB shmem:46144kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:9072kB unstable:0kB bounce:0kB
writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Mar 13 06:30:36 node1 kernel: [902670.980574] lowmem_reserve[]: 0 0 0 0
Mar 13 06:30:36 node1 kernel: [902670.980576] DMA: 15*4kB 15*8kB 9*16kB 22*32kB 7*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 1*2048kB 0*4096kB = 3524kB
Mar 13 06:30:36 node1 kernel: [902670.980583] Normal: 3507*4kB 3158*8kB 39*16kB 8*32kB 1*64kB 1*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 40364kB
Mar 13 06:30:36 node1 kernel: [902670.980589] HighMem: 47*4kB 57*8kB 23*16kB 12*32kB 2055*64kB 2550*128kB 704*256kB 104*512kB 13*1024kB 0*2048kB
0*4096kB = 706100kB
Mar 13 06:30:36 node1 kernel: [902670.980595] 612188 total pagecache pages
Mar 13 06:30:36 node1 kernel: [902670.980597] 160 pages in swap cache
Mar 13 06:30:36 node1 kernel: [902670.980598] Swap cache stats: add 21929, delete 21769, find 243894/245286
Mar 13 06:30:36 node1 kernel: [902670.980599] Free swap  = 4880236kB
Mar 13 06:30:36 node1 kernel: [902670.980601] Total swap = 4883640kB
Mar 13 06:30:36 node1 kernel: [902670.989961] 1245184 pages RAM
Mar 13 06:30:36 node1 kernel: [902670.989963] 1017858 pages HighMem
Mar 13 06:30:36 node1 kernel: [902670.989964] 210949 pages reserved
Mar 13 06:30:36 node1 kernel: [902670.989965] 327213 pages shared
Mar 13 06:30:36 node1 kernel: [902670.989966] 642319 pages non-shared
Mar 13 06:30:42 node1 kernel: [902677.037409] __ratelimit: 80 callbacks suppressed
Mar 13 06:30:42 node1 kernel: [902677.037413] drbd1_receiver: page allocation failure. order:4, mode:0x4020
Mar 13 06:30:42 node1 kernel: [902677.037417] Pid: 25245, comm: drbd1_receiver Not tainted 2.6.32-5-vserver-686-bigmem #1
Mar 13 06:30:42 node1 kernel: [902677.037420] Call Trace:
Mar 13 06:30:42 node1 kernel: [902677.037428]  [<c109ad8c>] ? __alloc_pages_nodemask+0x484/0x4d9
Mar 13 06:30:42 node1 kernel: [902677.037433]  [<c10255dc>] ? kunmap_atomic+0x48/0x57
Mar 13 06:30:42 node1 kernel: [902677.037436]  [<c109aded>] ? __get_free_pages+0xc/0x17
Mar 13 06:30:42 node1 kernel: [902677.037440]  [<c10bf261>] ? __kmalloc+0x30/0x128
Mar 13 06:30:42 node1 kernel: [902677.037444]  [<c10255dc>] ? kunmap_atomic+0x48/0x57
Mar 13 06:30:42 node1 kernel: [902677.037448]  [<c11ee2f4>] ? pskb_expand_head+0x4f/0x157
Mar 13 06:30:42 node1 kernel: [902677.037452]  [<c11ee74b>] ? __pskb_pull_tail+0x40/0x200
Mar 13 06:30:42 node1 kernel: [902677.037455]  [<c1204d46>] ? __qdisc_run+0x90/0xaa
Mar 13 06:30:42 node1 kernel: [902677.037459]  [<c11f555e>] ? dev_queue_xmit+0xe4/0x38e
Mar 13 06:30:42 node1 kernel: [902677.037463]  [<c1216cea>] ? ip_finish_output2+0x187/0x1c2
Mar 13 06:30:42 node1 kernel: [902677.037466]  [<c12161ed>] ? ip_local_out+0x15/0x17
Mar 13 06:30:42 node1 kernel: [902677.037469]  [<c12169cc>] ? ip_queue_xmit+0x31d/0x378
Mar 13 06:30:42 node1 kernel: [902677.037473]  [<c102ec17>] ? find_busiest_group+0x2e9/0x6e6
Mar 13 06:30:42 node1 kernel: [902677.037477]  [<c1099783>] ? free_one_page+0x52/0x5e
Mar 13 06:30:42 node1 kernel: [902677.037481]  [<c1225c52>] ? tcp_transmit_skb+0x595/0x5cc
Mar 13 06:30:42 node1 kernel: [902677.037485]  [<c1227cb5>] ? tcp_write_xmit+0x7a3/0x874
Mar 13 06:30:42 node1 kernel: [902677.037488]  [<c12236e4>] ? tcp_ack+0x1611/0x1802
Mar 13 06:30:42 node1 kernel: [902677.037492]  [<c1224fdf>] ? tcp_established_options+0x1d/0x8b
Mar 13 06:30:42 node1 kernel: [902677.037495]  [<c12252a3>] ? tcp_current_mss+0x38/0x53
Mar 13 06:30:42 node1 kernel: [902677.037498]  [<c1227dcc>] ? __tcp_push_pending_frames+0x1e/0x50
Mar 13 06:30:42 node1 kernel: [902677.037502]  [<c12238f0>] ? tcp_data_snd_check+0x1b/0xd2
Mar 13 06:30:42 node1 kernel: [902677.037505]  [<c1223f8f>] ? tcp_rcv_established+0xd2/0x62c
Mar 13 06:30:42 node1 kernel: [902677.037509]  [<c128afe9>] ? schedule_timeout+0x20/0xb0
Mar 13 06:30:42 node1 kernel: [902677.037513]  [<c122a9ae>] ? tcp_v4_do_rcv+0x15f/0x2cf
Mar 13 06:30:42 node1 kernel: [902677.037516]  [<c11eabce>] ? sk_wait_data+0x99/0xa2
Mar 13 06:30:42 node1 kernel: [902677.037520]  [<c121bf64>] ? tcp_prequeue_process+0x57/0x6b
Mar 13 06:30:42 node1 kernel: [902677.037524]  [<c121cf66>] ? tcp_recvmsg+0x441/0x8e5
Mar 13 06:30:42 node1 kernel: [902677.037527]  [<c11e9cdf>] ? sock_common_recvmsg+0x2f/0x45
Mar 13 06:30:42 node1 kernel: [902677.037530]  [<c11e9cdf>] ? sock_common_recvmsg+0x2f/0x45
Mar 13 06:30:42 node1 kernel: [902677.037535]  [<c11e7fc3>] ? __sock_recvmsg+0x50/0xbb
Mar 13 06:30:42 node1 kernel: [902677.037538]  [<c11e855d>] ? sock_recvmsg+0x9d/0xb6
Mar 13 06:30:42 node1 kernel: [902677.037543]  [<c104b08a>] ? autoremove_wake_function+0x0/0x2d
Mar 13 06:30:42 node1 kernel: [902677.037547]  [<c113c658>] ? __make_request+0x364/0x3d9
Mar 13 06:30:42 node1 kernel: [902677.037556]  [<f9067ddf>] ? drbd_recv+0x62/0x106 [drbd]
Mar 13 06:30:42 node1 kernel: [902677.037563]  [<f9067e95>] ? drbd_recv_header+0x12/0xa5 [drbd]
Mar 13 06:30:42 node1 kernel: [902677.037570]  [<f906e669>] ? drbdd_init+0xf3/0x245 [drbd]
Mar 13 06:30:42 node1 kernel: [902677.037577]  [<f9079bf7>] ? drbd_thread_setup+0x1f/0xcc [drbd]
Mar 13 06:30:42 node1 kernel: [902677.037584]  [<f9079bd8>] ? drbd_thread_setup+0x0/0xcc [drbd]
Mar 13 06:30:42 node1 kernel: [902677.037587]  [<c104ae58>] ? kthread+0x61/0x66
Mar 13 06:30:42 node1 kernel: [902677.037590]  [<c104adf7>] ? kthread+0x0/0x66
Mar 13 06:30:42 node1 kernel: [902677.037594]  [<c1008d87>] ? kernel_thread_helper+0x7/0x10
Mar 13 06:30:42 node1 kernel: [902677.037596] Mem-Info:
Mar 13 06:30:42 node1 kernel: [902677.037598] DMA per-cpu:
Mar 13 06:30:42 node1 kernel: [902677.037600] CPU    0: hi:    0, btch:   1 usd:   0
Mar 13 06:30:42 node1 kernel: [902677.037602] CPU    1: hi:    0, btch:   1 usd:   0
Mar 13 06:30:42 node1 kernel: [902677.037604] Normal per-cpu:
Mar 13 06:30:42 node1 kernel: [902677.037606] CPU    0: hi:  186, btch:  31 usd:   6
Mar 13 06:30:42 node1 kernel: [902677.037609] CPU    1: hi:  186, btch:  31 usd:  30
Mar 13 06:30:42 node1 kernel: [902677.037610] HighMem per-cpu:
Mar 13 06:30:42 node1 kernel: [902677.037612] CPU    0: hi:  186, btch:  31 usd:   0
Mar 13 06:30:42 node1 kernel: [902677.037615] CPU    1: hi:  186, btch:  31 usd:   0
Mar 13 06:30:42 node1 kernel: [902677.037619] active_anon:137310 inactive_anon:39519 isolated_anon:2
Mar 13 06:30:42 node1 kernel: [902677.037621]  active_file:223620 inactive_file:372790 isolated_file:36
Mar 13 06:30:42 node1 kernel: [902677.037622]  unevictable:4083 dirty:3716 writeback:1 unstable:0
Mar 13 06:30:42 node1 kernel: [902677.037623]  free:190562 slab_reclaimable:49977 slab_unreclaimable:4122
Mar 13 06:30:42 node1 kernel: [902677.037624]  mapped:23637 shmem:11538 pagetables:2294 bounce:158
Mar 13 06:30:42 node1 kernel: [902677.037632] DMA free:3524kB min:64kB low:80kB high:96kB active_anon:0kB inactive_anon:0kB active_file:2728kB
inactive_file:6180kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15868kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB
slab_reclaimable:3512kB slab_unreclaimable:8kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0
all_unreclaimable? no
Mar 13 06:30:42 node1 kernel: [902677.037637] lowmem_reserve[]: 0 865 4030 4030
Mar 13 06:30:42 node1 kernel: [902677.037650] Normal free:41960kB min:3728kB low:4660kB high:5592kB active_anon:4276kB inactive_anon:15768kB
active_file:199060kB inactive_file:354760kB unevictable:0kB isolated(anon):8kB isolated(file):144kB present:885944kB mlocked:0kB dirty:732kB
writeback:4kB mapped:24kB shmem:8kB slab_reclaimable:196396kB slab_unreclaimable:16384kB kernel_stack:2616kB pagetables:104kB unstable:0kB
bounce:632kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Mar 13 06:30:42 node1 kernel: [902677.037654] lowmem_reserve[]: 0 0 25321 25321
Mar 13 06:30:42 node1 kernel: [902677.037659] HighMem free:716764kB min:512kB low:3920kB high:7332kB active_anon:544964kB inactive_anon:142308kB
active_file:692692kB inactive_file:1130220kB unevictable:16332kB isolated(anon):0kB isolated(file):0kB present:3241132kB mlocked:16332kB dirty:14132kB
writeback:0kB mapped:94524kB shmem:46144kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:9072kB unstable:0kB bounce:0kB
writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Mar 13 06:30:42 node1 kernel: [902677.037664] lowmem_reserve[]: 0 0 0 0
Mar 13 06:30:42 node1 kernel: [902677.037666] DMA: 15*4kB 15*8kB 9*16kB 22*32kB 7*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 1*2048kB 0*4096kB = 3524kB
Mar 13 06:30:42 node1 kernel: [902677.037673] Normal: 3610*4kB 3223*8kB 69*16kB 16*32kB 1*64kB 1*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB =
42032kB
Mar 13 06:30:42 node1 kernel: [902677.037679] HighMem: 107*4kB 560*8kB 567*16kB 361*32kB 1846*64kB 2550*128kB 704*256kB 104*512kB 13*1024kB 0*2048kB
0*4096kB = 716860kB
Mar 13 06:30:42 node1 kernel: [902677.037686] 609506 total pagecache pages
Mar 13 06:30:42 node1 kernel: [902677.037687] 169 pages in swap cache
Mar 13 06:30:42 node1 kernel: [902677.037688] Swap cache stats: add 21938, delete 21769, find 243894/245286
Mar 13 06:30:42 node1 kernel: [902677.037690] Free swap  = 4880200kB
Mar 13 06:30:42 node1 kernel: [902677.037691] Total swap = 4883640kB
Mar 13 06:30:42 node1 kernel: [902677.047173] 1245184 pages RAM
Mar 13 06:30:42 node1 kernel: [902677.047174] 1017858 pages HighMem
Mar 13 06:30:42 node1 kernel: [902677.047175] 210949 pages reserved
Mar 13 06:30:42 node1 kernel: [902677.047176] 328890 pages shared
Mar 13 06:30:42 node1 kernel: [902677.047177] 637938 pages non-shared
Mar 13 06:30:42 node1 kernel: [902677.047193] drbd1_receiver: page allocation failure. order:4, mode:0x4020
Mar 13 06:30:42 node1 kernel: [902677.047196] Pid: 25245, comm: drbd1_receiver Not tainted 2.6.32-5-vserver-686-bigmem #1
Mar 13 06:30:42 node1 kernel: [902677.047198] Call Trace:
Mar 13 06:30:42 node1 kernel: [902677.047204]  [<c109ad8c>] ? __alloc_pages_nodemask+0x484/0x4d9
Mar 13 06:30:42 node1 kernel: [902677.047208]  [<c10255dc>] ? kunmap_atomic+0x48/0x57
Mar 13 06:30:42 node1 kernel: [902677.047210]  [<c109aded>] ? __get_free_pages+0xc/0x17
Mar 13 06:30:42 node1 kernel: [902677.047213]  [<c10bf261>] ? __kmalloc+0x30/0x128
Mar 13 06:30:42 node1 kernel: [902677.047215]  [<c10255dc>] ? kunmap_atomic+0x48/0x57
Mar 13 06:30:42 node1 kernel: [902677.047218]  [<c11ee2f4>] ? pskb_expand_head+0x4f/0x157
Mar 13 06:30:42 node1 kernel: [902677.047221]  [<c11ee74b>] ? __pskb_pull_tail+0x40/0x200
Mar 13 06:30:42 node1 kernel: [902677.047223]  [<c11eb077>] ? sock_wfree+0x17/0x4b
Mar 13 06:30:42 node1 kernel: [902677.047226]  [<c11f555e>] ? dev_queue_xmit+0xe4/0x38e
Mar 13 06:30:42 node1 kernel: [902677.047229]  [<c1216cea>] ? ip_finish_output2+0x187/0x1c2
Mar 13 06:30:42 node1 kernel: [902677.047231]  [<c12161ed>] ? ip_local_out+0x15/0x17
Mar 13 06:30:42 node1 kernel: [902677.047233]  [<c12169cc>] ? ip_queue_xmit+0x31d/0x378
Mar 13 06:30:42 node1 kernel: [902677.047236]  [<c102ec17>] ? find_busiest_group+0x2e9/0x6e6
Mar 13 06:30:42 node1 kernel: [902677.047238]  [<c1099783>] ? free_one_page+0x52/0x5e
Mar 13 06:30:42 node1 kernel: [902677.047241]  [<c1225c52>] ? tcp_transmit_skb+0x595/0x5cc
Mar 13 06:30:42 node1 kernel: [902677.047243]  [<c1227cb5>] ? tcp_write_xmit+0x7a3/0x874
Mar 13 06:30:42 node1 kernel: [902677.047245]  [<c12236e4>] ? tcp_ack+0x1611/0x1802
Mar 13 06:30:42 node1 kernel: [902677.047248]  [<c1224fdf>] ? tcp_established_options+0x1d/0x8b
Mar 13 06:30:42 node1 kernel: [902677.047250]  [<c12252a3>] ? tcp_current_mss+0x38/0x53
Mar 13 06:30:42 node1 kernel: [902677.047252]  [<c1227dcc>] ? __tcp_push_pending_frames+0x1e/0x50
Mar 13 06:30:42 node1 kernel: [902677.047254]  [<c12238f0>] ? tcp_data_snd_check+0x1b/0xd2
Mar 13 06:30:42 node1 kernel: [902677.047256]  [<c1224493>] ? tcp_rcv_established+0x5d6/0x62c
Mar 13 06:30:42 node1 kernel: [902677.047260]  [<c128afe9>] ? schedule_timeout+0x20/0xb0
Mar 13 06:30:42 node1 kernel: [902677.047262]  [<c122a9ae>] ? tcp_v4_do_rcv+0x15f/0x2cf
Mar 13 06:30:42 node1 kernel: [902677.047264]  [<c11eabce>] ? sk_wait_data+0x99/0xa2
Mar 13 06:30:42 node1 kernel: [902677.047267]  [<c121bf64>] ? tcp_prequeue_process+0x57/0x6b
Mar 13 06:30:42 node1 kernel: [902677.047269]  [<c121cf66>] ? tcp_recvmsg+0x441/0x8e5
Mar 13 06:30:42 node1 kernel: [902677.047272]  [<c11e9cdf>] ? sock_common_recvmsg+0x2f/0x45
Mar 13 06:30:42 node1 kernel: [902677.047274]  [<c11e9cdf>] ? sock_common_recvmsg+0x2f/0x45
Mar 13 06:30:42 node1 kernel: [902677.047277]  [<c11e7fc3>] ? __sock_recvmsg+0x50/0xbb
Mar 13 06:30:42 node1 kernel: [902677.047279]  [<c11e855d>] ? sock_recvmsg+0x9d/0xb6
Mar 13 06:30:42 node1 kernel: [902677.047282]  [<c104b08a>] ? autoremove_wake_function+0x0/0x2d
Mar 13 06:30:42 node1 kernel: [902677.047286]  [<c113c658>] ? __make_request+0x364/0x3d9
Mar 13 06:30:42 node1 kernel: [902677.047291]  [<f9067ddf>] ? drbd_recv+0x62/0x106 [drbd]
Mar 13 06:30:42 node1 kernel: [902677.047296]  [<f9067e95>] ? drbd_recv_header+0x12/0xa5 [drbd]
Mar 13 06:30:42 node1 kernel: [902677.047301]  [<f906e669>] ? drbdd_init+0xf3/0x245 [drbd]
Mar 13 06:30:42 node1 kernel: [902677.047305]  [<f9079bf7>] ? drbd_thread_setup+0x1f/0xcc [drbd]
Mar 13 06:30:42 node1 kernel: [902677.047310]  [<f9079bd8>] ? drbd_thread_setup+0x0/0xcc [drbd]
Mar 13 06:30:42 node1 kernel: [902677.047312]  [<c104ae58>] ? kthread+0x61/0x66
Mar 13 06:30:42 node1 kernel: [902677.047314]  [<c104adf7>] ? kthread+0x0/0x66
Mar 13 06:30:42 node1 kernel: [902677.047317]  [<c1008d87>] ? kernel_thread_helper+0x7/0x10
Mar 13 06:30:42 node1 kernel: [902677.047319] Mem-Info:
Mar 13 06:30:42 node1 kernel: [902677.047320] DMA per-cpu:
Mar 13 06:30:42 node1 kernel: [902677.047321] CPU    0: hi:    0, btch:   1 usd:   0
Mar 13 06:30:42 node1 kernel: [902677.047323] CPU    1: hi:    0, btch:   1 usd:   0
Mar 13 06:30:42 node1 kernel: [902677.047324] Normal per-cpu:
Mar 13 06:30:42 node1 kernel: [902677.047326] CPU    0: hi:  186, btch:  31 usd:   5
Mar 13 06:30:42 node1 kernel: [902677.047327] CPU    1: hi:  186, btch:  31 usd:  30
Mar 13 06:30:42 node1 kernel: [902677.047328] HighMem per-cpu:
Mar 13 06:30:42 node1 kernel: [902677.047330] CPU    0: hi:  186, btch:  31 usd:  25
Mar 13 06:30:42 node1 kernel: [902677.047331] CPU    1: hi:  186, btch:  31 usd:   0
Mar 13 06:30:42 node1 kernel: [902677.047334] active_anon:137310 inactive_anon:39519 isolated_anon:2
Mar 13 06:30:42 node1 kernel: [902677.047335]  active_file:223620 inactive_file:372790 isolated_file:36
Mar 13 06:30:42 node1 kernel: [902677.047336]  unevictable:4083 dirty:3716 writeback:1 unstable:0
Mar 13 06:30:42 node1 kernel: [902677.047337]  free:190531 slab_reclaimable:49977 slab_unreclaimable:4098
Mar 13 06:30:42 node1 kernel: [902677.047338]  mapped:23637 shmem:11538 pagetables:2294 bounce:158
Mar 13 06:30:42 node1 kernel: [902677.047342] DMA free:3524kB min:64kB low:80kB high:96kB active_anon:0kB inactive_anon:0kB active_file:2728kB
inactive_file:6180kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15868kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB
slab_reclaimable:3512kB slab_unreclaimable:8kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0
all_unreclaimable? no
Mar 13 06:30:42 node1 kernel: [902677.047346] lowmem_reserve[]: 0 865 4030 4030
Mar 13 06:30:42 node1 kernel: [902677.047352] Normal free:41960kB min:3728kB low:4660kB high:5592kB active_anon:4276kB inactive_anon:15768kB
active_file:199060kB inactive_file:354760kB unevictable:0kB isolated(anon):8kB isolated(file):144kB present:885944kB mlocked:0kB dirty:732kB
writeback:4kB mapped:24kB shmem:8kB slab_reclaimable:196396kB slab_unreclaimable:16384kB kernel_stack:2616kB pagetables:104kB unstable:0kB
bounce:632kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Mar 13 06:30:42 node1 kernel: [902677.047356] lowmem_reserve[]: 0 0 25321 25321
Mar 13 06:30:42 node1 kernel: [902677.047362] HighMem free:716640kB min:512kB low:3920kB high:7332kB active_anon:544964kB inactive_anon:142308kB
active_file:692692kB inactive_file:1130220kB unevictable:16332kB isolated(anon):0kB isolated(file):0kB present:3241132kB mlocked:16332kB dirty:14132kB
writeback:0kB mapped:94524kB shmem:46144kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:9072kB unstable:0kB bounce:0kB
writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Mar 13 06:30:42 node1 kernel: [902677.047366] lowmem_reserve[]: 0 0 0 0
Mar 13 06:30:42 node1 kernel: [902677.047369] DMA: 15*4kB 15*8kB 9*16kB 22*32kB 7*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 1*2048kB 0*4096kB = 3524kB
Mar 13 06:30:42 node1 kernel: [902677.047375] Normal: 3610*4kB 3223*8kB 69*16kB 16*32kB 1*64kB 1*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB =
42032kB
Mar 13 06:30:42 node1 kernel: [902677.047382] HighMem: 106*4kB 545*8kB 567*16kB 361*32kB 1846*64kB 2550*128kB 704*256kB 104*512kB 13*1024kB 0*2048kB
0*4096kB = 716736kB
Mar 13 06:30:42 node1 kernel: [902677.047389] 609506 total pagecache pages
Mar 13 06:30:42 node1 kernel: [902677.047390] 169 pages in swap cache
Mar 13 06:30:42 node1 kernel: [902677.047391] Swap cache stats: add 21938, delete 21769, find 243894/245286
Mar 13 06:30:42 node1 kernel: [902677.047393] Free swap  = 4880200kB
Mar 13 06:30:42 node1 kernel: [902677.047394] Total swap = 4883640kB
Mar 13 06:30:42 node1 kernel: [902677.056949] 1245184 pages RAM
Mar 13 06:30:42 node1 kernel: [902677.056951] 1017858 pages HighMem
Mar 13 06:30:42 node1 kernel: [902677.056952] 210949 pages reserved
Mar 13 06:30:42 node1 kernel: [902677.056953] 328766 pages shared
Mar 13 06:30:42 node1 kernel: [902677.056954] 637885 pages non-shared

Can someone please direct me where to look?

-- 
pozdrawiam,
Konrad Korzeniowski




More information about the drbd-user mailing list