Note: "permalinks" may not be as permanent as we would like,
direct links of old sources may well be a few messages off.
On Fri, Jan 16, 2009 at 12:35:03PM -0000, Robert Dunkley wrote: > Hi Everyone, > > I'm running DRBD 8.2.6 on a Centos 5.2 X64 box with Xen 3.30 installed. > The disks are partitioned with LVM and then DRBD runs on top. One of the > disks keeps erroring as shown below. The secondary system shows the > error, no errors appear on the primary. please try to avoid line wraps when pasting this kind of stuff, makes it much more readable. > Any ideas what this could be? sure. your (Xen) tcp stack is out of memory, it cannot even send one more single message out. in fact, according to the Mem-info given below, your whole box is seriously out of memory. may or may not be related to Xen. maybe you can workaround that by tuning some memory settings, or some tcp memory setting, or some xen setting. probably you should just put more ram into the boxes. 1GB seems a bit tight. > Jan 16 12:19:12 mrtickle kernel: drbd13_receiver: page allocation failure. order:4, mode:0x10 > Jan 16 12:19:12 mrtickle kernel: > Jan 16 12:19:12 mrtickle kernel: Call Trace: > Jan 16 12:19:12 mrtickle kernel: [<ffffffff8020f662>] __alloc_pages+0x2b5/0x2ce > Jan 16 12:19:12 mrtickle kernel: [<ffffffff8025e80e>] cache_alloc_refill+0x269/0x4ba > Jan 16 12:19:12 mrtickle kernel: [<ffffffff8020af77>] kmem_cache_alloc+0x50/0x6d > Jan 16 12:19:12 mrtickle kernel: [<ffffffff80236337>] alloc_skb_from_cache+0x52/0x13c > Jan 16 12:19:12 mrtickle kernel: [<ffffffff8022669a>] tcp_sendmsg+0x184/0xafb > Jan 16 12:19:12 mrtickle kernel: [<ffffffff80256d13>] sock_sendmsg+0xf3/0x110 > Jan 16 12:19:12 mrtickle kernel: [<ffffffff8029bade>] autoremove_wake_function+0x0/0x2e > Jan 16 12:19:12 mrtickle kernel: [<ffffffff80231986>] sock_recvmsg+0x101/0x120 > Jan 16 12:19:12 mrtickle kernel: [<ffffffff80232e4f>] sock_common_recvmsg+0x2d/0x43 > Jan 16 12:19:12 mrtickle kernel: [<ffffffff8029bade>] autoremove_wake_function+0x0/0x2e > Jan 16 12:19:12 mrtickle kernel: [<ffffffff8029bade>] autoremove_wake_function+0x0/0x2e > Jan 16 12:19:12 mrtickle kernel: [<ffffffff8040382e>] kernel_sendmsg+0x35/0x47 > Jan 16 12:19:12 mrtickle kernel: [<ffffffff8869288c>] :drbd:drbd_send+0x8e/0x167 > Jan 16 12:19:12 mrtickle kernel: [<ffffffff80262d88>] thread_return+0x5d/0xf7 > Jan 16 12:19:12 mrtickle kernel: [<ffffffff88692c15>] :drbd:_drbd_send_cmd+0x110/0x18d > Jan 16 12:19:12 mrtickle kernel: [<ffffffff88692e24>] :drbd:drbd_send_cmd+0x84/0xc0 > Jan 16 12:19:12 mrtickle kernel: [<ffffffff886949a0>] :drbd:drbd_send_b_ack+0x23/0x28 > Jan 16 12:19:12 mrtickle kernel: [<ffffffff8868827e>] :drbd:receive_Barrier_no_tcq+0x325/0x37f > Jan 16 12:19:12 mrtickle kernel: [<ffffffff8029bade>] autoremove_wake_function+0x0/0x2e > Jan 16 12:19:12 mrtickle kernel: [<ffffffff88682c4b>] :drbd:drbdd+0x77/0x152 > Jan 16 12:19:12 mrtickle kernel: [<ffffffff88685870>] :drbd:drbdd_init+0xea/0x1dc > Jan 16 12:19:12 mrtickle kernel: [<ffffffff8869532a>] :drbd:drbd_thread_setup+0xa2/0x18b > Jan 16 12:19:12 mrtickle kernel: [<ffffffff80260b24>] child_rip+0xa/0x12 > Jan 16 12:19:12 mrtickle kernel: [<ffffffff88695288>] :drbd:drbd_thread_setup+0x0/0x18b > Jan 16 12:19:12 mrtickle kernel: [<ffffffff80260b1a>] child_rip+0x0/0x12 > Jan 16 12:19:12 mrtickle kernel: > Jan 16 12:19:12 mrtickle kernel: Mem-info: > Jan 16 12:19:12 mrtickle kernel: DMA per-cpu: > Jan 16 12:19:12 mrtickle kernel: cpu 0 hot: high 186, batch 31 used:166 > Jan 16 12:19:12 mrtickle kernel: cpu 0 cold: high 62, batch 15 used:59 > Jan 16 12:19:12 mrtickle kernel: cpu 1 hot: high 186, batch 31 used:0 > Jan 16 12:19:12 mrtickle kernel: cpu 1 cold: high 62, batch 15 used:14 > Jan 16 12:19:12 mrtickle kernel: cpu 2 hot: high 186, batch 31 used:19 > Jan 16 12:19:12 mrtickle kernel: cpu 2 cold: high 62, batch 15 used:53 > Jan 16 12:19:12 mrtickle kernel: cpu 3 hot: high 186, batch 31 used:8 > Jan 16 12:19:12 mrtickle kernel: cpu 3 cold: high 62, batch 15 used:53 > Jan 16 12:19:12 mrtickle kernel: cpu 4 hot: high 186, batch 31 used:6 > Jan 16 12:19:12 mrtickle kernel: cpu 4 cold: high 62, batch 15 used:55 > Jan 16 12:19:12 mrtickle kernel: cpu 5 hot: high 186, batch 31 used:56 > Jan 16 12:19:12 mrtickle kernel: cpu 5 cold: high 62, batch 15 used:43 > Jan 16 12:19:12 mrtickle kernel: cpu 6 hot: high 186, batch 31 used:14 > Jan 16 12:19:12 mrtickle kernel: cpu 6 cold: high 62, batch 15 used:58 > Jan 16 12:19:12 mrtickle kernel: cpu 7 hot: high 186, batch 31 used:1 > Jan 16 12:19:12 mrtickle kernel: cpu 7 cold: high 62, batch 15 used:48 > Jan 16 12:19:12 mrtickle kernel: DMA32 per-cpu: empty > Jan 16 12:19:12 mrtickle kernel: Normal per-cpu: empty > Jan 16 12:19:12 mrtickle kernel: HighMem per-cpu: empty > Jan 16 12:19:12 mrtickle kernel: Free pages: 4100kB (0kB HighMem) > Jan 16 12:19:12 mrtickle kernel: Active:82078 inactive:114665 dirty:88725 writeback:4 unstable:0 free:1025 slab:13289 mapped-file:3191 mapped-anon:32051 pagetables:5857 > Jan 16 12:19:12 mrtickle kernel: DMA free:4100kB min:4108kB low:5132kB high:6160kB active:328312kB inactive:458660kB present:1056768kB pages_scanned:222 all_unreclaimable? no > Jan 16 12:19:12 mrtickle kernel: lowmem_reserve[]: 0 0 0 0 > Jan 16 12:19:12 mrtickle kernel: DMA32 free:0kB min:0kB low:0kB high:0kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no > Jan 16 12:19:12 mrtickle kernel: lowmem_reserve[]: 0 0 0 0 > Jan 16 12:19:12 mrtickle kernel: Normal free:0kB min:0kB low:0kB high:0kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no > Jan 16 12:19:12 mrtickle kernel: lowmem_reserve[]: 0 0 0 0 > Jan 16 12:19:12 mrtickle kernel: HighMem free:0kB min:128kB low:128kB high:128kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no > Jan 16 12:19:12 mrtickle kernel: lowmem_reserve[]: 0 0 0 0 > Jan 16 12:19:12 mrtickle kernel: DMA: 67*4kB 7*8kB 8*16kB 2*32kB 4*64kB 2*128kB 0*256kB 2*512kB 0*1024kB 1*2048kB 0*4096kB = 4100kB > Jan 16 12:19:12 mrtickle kernel: DMA32: empty > Jan 16 12:19:12 mrtickle kernel: Normal: empty > Jan 16 12:19:12 mrtickle kernel: HighMem: empty > Jan 16 12:19:12 mrtickle kernel: 164877 pagecache pages > Jan 16 12:19:12 mrtickle kernel: Swap cache: add 58, delete 58, find 0/0, race 0+0 > Jan 16 12:19:12 mrtickle kernel: Free swap = 2031428kB > Jan 16 12:19:12 mrtickle kernel: Total swap = 2031608kB > Jan 16 12:19:12 mrtickle kernel: Free swap: 2031428kB > Jan 16 12:19:12 mrtickle kernel: 264192 pages of RAM > Jan 16 12:19:12 mrtickle kernel: 25356 reserved pages > Jan 16 12:19:12 mrtickle kernel: 56494 pages shared > Jan 16 12:19:12 mrtickle kernel: 0 pages swap cached -- : Lars Ellenberg : LINBIT | Your Way to High Availability : DRBD/HA support and consulting http://www.linbit.com DRBD® and LINBIT® are registered trademarks of LINBIT, Austria. __ please don't Cc me, but send to list -- I'm subscribed