Note: "permalinks" may not be as permanent as we would like,
direct links of old sources may well be a few messages off.
Figured this out. It caused by huge buffer size, which is kind of like cache, but with disk meta data. The kernel won't release buffered memory unless the kernel tries to get more memory, and *buffered* memory accumulates over time. I ran a program that allocates a git chunk of memory and the buffer size went down to less then 1MB. On Tue, May 21, 2013 at 12:14 PM, Lin Zhao <lin at groupon.com> wrote: > I've been running DRBD set up for 3 months, and recently noticed high > kernel memory usage on the secondary machines. > > The secondary machine runs very light user applications, but total memory > usage reaches as much as 60G. > > Is there a known issue with kernel message leak? Attaching top, slabtop > and meminfo from my backup machine. You can see that the processes from * > top* show very light RES size, but the system memory usage reaches 63 G. > Can you identify something obvious? > > top: > last pid: 5836; load avg: 0.11, 0.17, 0.12; up 250+16:35:23 > > 19:12:32 > 326 processes: 1 running, 325 sleeping > CPU states: 0.0% user, 0.0% nice, 0.0% system, 100% idle, 0.0% iowait > Kernel: 106 ctxsw, 1019 intr > Memory: 63G used, 426M free, 52G buffers, 179M cached > Swap: 116K used, 8000M free > > PID USERNAME THR PRI NICE SIZE RES SHR STATE TIME CPU COMMAND > 25785 ganglia 1 15 0 140M 8460K 3476K sleep 88:42 0.00% gmond > 4730 root 1 16 0 88M 3368K 2636K sleep 0:00 0.00% sshd > 4732 lin 1 15 0 88M 1836K 1084K sleep 0:00 0.00% sshd > 4733 lin 1 16 0 65M 1596K 1272K sleep 0:00 0.00% bash > 7523 root 1 15 0 65M 1596K 1272K sleep 0:00 0.00% bash > 5500 root 1 15 0 61M 1208K 644K sleep 2:20 0.00% sshd > 5834 root 1 15 0 61M 848K 336K sleep 0:00 0.00% crond > 8785 root 1 16 0 61M 1024K 516K sleep 0:03 0.00% crond > 7493 root 1 15 0 51M 1372K 1036K sleep 0:00 0.00% login > 5066 root 3 20 0 28M 576K 448K sleep 0:00 0.00% > brcm_iscsiuio > 8886 root 1 15 0 23M 1984K 1464K sleep 0:00 0.00% ntpd > 1798 root 1 11 -4 12M 776K 456K sleep 0:00 0.00% udevd > 5072 root 1 5 -10 12M 4452K 3164K sleep 0:00 0.00% iscsid > 5071 root 1 18 0 12M 652K 416K sleep 0:00 0.00% iscsid > 5718 lin 1 15 0 11M 1152K 848K run 0:00 0.00% top > 12349 root 1 15 0 11M 1532K 612K sleep 2:54 0.00% syslogd > 1 root 1 15 0 10M 752K 632K sleep 4:17 0.00% init > 5835 root 1 19 0 8688K 1072K 924K sleep 0:00 0.00% sh > 7301 root 1 19 0 3808K 532K 448K sleep 0:00 0.00% mingetty > 7300 root 1 18 0 3808K 532K 448K sleep 0:00 0.00% mingetty > 7299 root 1 17 0 3808K 532K 448K sleep 0:00 0.00% mingetty > 7298 root 1 16 0 3808K 532K 448K sleep 0:00 0.00% mingetty > 7302 root 1 18 0 3808K 528K 448K sleep 0:00 0.00% mingetty > 7303 root 1 18 0 3808K 528K 448K sleep 0:00 0.00% mingetty > 5836 root 1 19 0 3808K 484K 408K sleep 0:00 0.00% sleep > 1744 root 1 10 -5 0K 0K 0K sleep 649:36 0.00% > md2_raid10 > 6586 root 1 15 0 0K 0K 0K sleep 227:57 0.00% > drbd1_receiver > 6587 root 1 -3 0 0K 0K 0K sleep 72:20 0.00% > drbd1_asender > 1740 root 1 10 -5 0K 0K 0K sleep 64:18 0.00% > md1_raid10 > 1750 root 1 10 -5 0K 0K 0K sleep 16:02 0.00% > kjournald > > slabtop: > Active / Total Objects (% used) : 108378294 / 108636165 (99.8%) > Active / Total Slabs (% used) : 2746709 / 2746710 (100.0%) > Active / Total Caches (% used) : 100 / 150 (66.7%) > Active / Total Size (% used) : 10273556.84K / 10298936.27K (99.8%) > Minimum / Average / Maximum Object : 0.02K / 0.09K / 128.00K > > OBJS ACTIVE USE OBJ SIZE SLABS OBJ/SLAB CACHE SIZE NAME > > 108327280 108091540 20% 0.09K 2708182 40 10832728K buffer_head > 228606 228226 99% 0.52K 32658 7 130632K radix_tree_node > 9856 9832 99% 0.09K 224 44 896K sysfs_dir_cache > 7847 3871 49% 0.06K 133 59 532K size-64 > 7596 5889 77% 0.21K 422 18 1688K dentry_cache > 6300 5208 82% 0.12K 210 30 840K size-128 > 4368 3794 86% 0.03K 39 112 156K size-32 > 3150 2793 88% 0.25K 210 15 840K size-256 > 3068 2563 83% 0.06K 52 59 208K Acpi-Operand > 2904 1253 43% 0.17K 132 22 528K vm_area_struct > 2376 2342 98% 1.00K 594 4 2376K size-1024 > 2304 380 16% 0.02K 16 144 64K anon_vma > 2256 1852 82% 0.08K 47 48 188K selinux_inode_security > 2121 1943 91% 0.55K 303 7 1212K inode_cache > 1776 1463 82% 0.50K 222 8 888K size-512 > 1710 705 41% 0.25K 114 15 456K filp > 1698 1642 96% 0.58K 283 6 1132K proc_inode_cache > 1632 1606 98% 2.00K 816 2 3264K size-2048 > 1590 1147 72% 0.25K 106 15 424K skbuff_head_cache > 1584 324 20% 0.02K 11 144 44K numa_policy > 1180 359 30% 0.06K 20 59 80K delayacct_cache > 1140 1101 96% 0.74K 228 5 912K ext3_inode_cache > 1080 1049 97% 0.09K 27 40 108K drbd_ee > 1054 1024 97% 0.11K 31 34 124K drbd_req > 1010 339 33% 0.02K 5 202 20K biovec-1 > 1008 888 88% 0.03K 9 112 36K Acpi-Namespace > 944 335 35% 0.06K 16 59 64K pid > 650 514 79% 0.75K 130 5 520K shmem_inode_cache > 630 542 86% 0.12K 21 30 84K bio > 558 353 63% 0.81K 62 9 496K signal_cache > 496 496 100% 4.00K 496 1 1984K size-4096 > 410 351 85% 1.84K 205 2 820K task_struct > 399 355 88% 2.06K 133 3 1064K sighand_cache > 354 54 15% 0.06K 6 59 24K fs_cache > > memtop: > MemTotal: 65996216 kB > MemFree: 436188 kB > Buffers: 54272396 kB > Cached: 183784 kB > SwapCached: 0 kB > Active: 324660 kB > Inactive: 54143868 kB > HighTotal: 0 kB > HighFree: 0 kB > LowTotal: 65996216 kB > LowFree: 436188 kB > SwapTotal: 8192504 kB > SwapFree: 8192388 kB > Dirty: 0 kB > Writeback: 0 kB > AnonPages: 12320 kB > Mapped: 8312 kB > Slab: 10988324 kB > PageTables: 1584 kB > NFS_Unstable: 0 kB > Bounce: 0 kB > CommitLimit: 41190612 kB > Committed_AS: 44772 kB > VmallocTotal: 34359738367 kB > VmallocUsed: 267000 kB > VmallocChunk: 34359471059 kB > HugePages_Total: 0 > HugePages_Free: 0 > HugePages_Rsvd: 0 > Hugepagesize: 2048 kB > > -- > Lin Zhao > Project Lead of Messagebus > https://wiki.groupondev.com/Message_Bus > 3101 Park Blvd, Palo Alto, CA 94306 > -- Lin Zhao Project Lead of Messagebus https://wiki.groupondev.com/Message_Bus 3101 Park Blvd, Palo Alto, CA 94306 -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.linbit.com/pipermail/drbd-user/attachments/20130521/adbceabf/attachment.htm>