<div dir="ltr">I've been running DRBD set up for 3 months, and recently noticed high kernel memory usage on the secondary machines.<div><br></div><div>The secondary machine runs very light user applications, but total memory usage reaches as much as 60G.</div>
<div><br></div><div>Is there a known issue with kernel message leak? Attaching top, slabtop and meminfo from my backup machine. You can see that the processes from <i>top</i> show very light RES size, but the system memory usage reaches 63 G. Can you identify something obvious?</div>
<div><br></div><div>top:</div><div><div>last pid: 5836; load avg: 0.11, 0.17, 0.12; up 250+16:35:23 19:12:32</div>
<div>326 processes: 1 running, 325 sleeping</div><div>CPU states: 0.0% user, 0.0% nice, 0.0% system, 100% idle, 0.0% iowait</div><div>Kernel: 106 ctxsw, 1019 intr</div><div>Memory: 63G used, 426M free, 52G buffers, 179M cached</div>
<div>Swap: 116K used, 8000M free</div><div><br></div><div> PID USERNAME THR PRI NICE SIZE RES SHR STATE TIME CPU COMMAND</div><div>25785 ganglia 1 15 0 140M 8460K 3476K sleep 88:42 0.00% gmond</div>
<div> 4730 root 1 16 0 88M 3368K 2636K sleep 0:00 0.00% sshd</div><div> 4732 lin 1 15 0 88M 1836K 1084K sleep 0:00 0.00% sshd</div><div> 4733 lin 1 16 0 65M 1596K 1272K sleep 0:00 0.00% bash</div>
<div> 7523 root 1 15 0 65M 1596K 1272K sleep 0:00 0.00% bash</div><div> 5500 root 1 15 0 61M 1208K 644K sleep 2:20 0.00% sshd</div><div> 5834 root 1 15 0 61M 848K 336K sleep 0:00 0.00% crond</div>
<div> 8785 root 1 16 0 61M 1024K 516K sleep 0:03 0.00% crond</div><div> 7493 root 1 15 0 51M 1372K 1036K sleep 0:00 0.00% login</div><div> 5066 root 3 20 0 28M 576K 448K sleep 0:00 0.00% brcm_iscsiuio</div>
<div> 8886 root 1 15 0 23M 1984K 1464K sleep 0:00 0.00% ntpd</div><div> 1798 root 1 11 -4 12M 776K 456K sleep 0:00 0.00% udevd</div><div> 5072 root 1 5 -10 12M 4452K 3164K sleep 0:00 0.00% iscsid</div>
<div> 5071 root 1 18 0 12M 652K 416K sleep 0:00 0.00% iscsid</div><div> 5718 lin 1 15 0 11M 1152K 848K run 0:00 0.00% top</div><div>12349 root 1 15 0 11M 1532K 612K sleep 2:54 0.00% syslogd</div>
<div> 1 root 1 15 0 10M 752K 632K sleep 4:17 0.00% init</div><div> 5835 root 1 19 0 8688K 1072K 924K sleep 0:00 0.00% sh</div><div> 7301 root 1 19 0 3808K 532K 448K sleep 0:00 0.00% mingetty</div>
<div> 7300 root 1 18 0 3808K 532K 448K sleep 0:00 0.00% mingetty</div><div> 7299 root 1 17 0 3808K 532K 448K sleep 0:00 0.00% mingetty</div><div> 7298 root 1 16 0 3808K 532K 448K sleep 0:00 0.00% mingetty</div>
<div> 7302 root 1 18 0 3808K 528K 448K sleep 0:00 0.00% mingetty</div><div> 7303 root 1 18 0 3808K 528K 448K sleep 0:00 0.00% mingetty</div><div> 5836 root 1 19 0 3808K 484K 408K sleep 0:00 0.00% sleep</div>
<div> 1744 root 1 10 -5 0K 0K 0K sleep 649:36 0.00% md2_raid10</div><div> 6586 root 1 15 0 0K 0K 0K sleep 227:57 0.00% drbd1_receiver</div><div> 6587 root 1 -3 0 0K 0K 0K sleep 72:20 0.00% drbd1_asender</div>
<div> 1740 root 1 10 -5 0K 0K 0K sleep 64:18 0.00% md1_raid10</div><div> 1750 root 1 10 -5 0K 0K 0K sleep 16:02 0.00% kjournald</div><div><br></div><div style>slabtop:</div><div style>
<div>Active / Total Objects (% used) : 108378294 / 108636165 (99.8%)</div><div> Active / Total Slabs (% used) : 2746709 / 2746710 (100.0%)</div><div> Active / Total Caches (% used) : 100 / 150 (66.7%)</div><div>
Active / Total Size (% used) : 10273556.84K / 10298936.27K (99.8%)</div><div> Minimum / Average / Maximum Object : 0.02K / 0.09K / 128.00K</div><div><br></div><div> OBJS ACTIVE USE OBJ SIZE SLABS OBJ/SLAB CACHE SIZE NAME </div>
<div>108327280 108091540 20% 0.09K 2708182<span class="" style="white-space:pre"> </span>40 10832728K buffer_head</div><div>228606 228226 99% 0.52K 32658 7 130632K radix_tree_node</div><div> 9856 9832 99% 0.09K 224<span class="" style="white-space:pre"> </span> 44<span class="" style="white-space:pre"> </span> 896K sysfs_dir_cache</div>
<div> 7847 3871 49% 0.06K 133<span class="" style="white-space:pre"> </span> 59<span class="" style="white-space:pre"> </span> 532K size-64</div><div> 7596 5889 77% 0.21K 422<span class="" style="white-space:pre"> </span> 18<span class="" style="white-space:pre"> </span> 1688K dentry_cache</div>
<div> 6300 5208 82% 0.12K 210<span class="" style="white-space:pre"> </span> 30<span class="" style="white-space:pre"> </span> 840K size-128</div><div> 4368 3794 86% 0.03K 39<span class="" style="white-space:pre"> </span>112<span class="" style="white-space:pre"> </span> 156K size-32</div>
<div> 3150 2793 88% 0.25K 210<span class="" style="white-space:pre"> </span> 15<span class="" style="white-space:pre"> </span> 840K size-256</div><div> 3068 2563 83% 0.06K 52<span class="" style="white-space:pre"> </span> 59<span class="" style="white-space:pre"> </span> 208K Acpi-Operand</div>
<div> 2904 1253 43% 0.17K 132<span class="" style="white-space:pre"> </span> 22<span class="" style="white-space:pre"> </span> 528K vm_area_struct</div><div> 2376 2342 98% 1.00K 594 4<span class="" style="white-space:pre"> </span> 2376K size-1024</div>
<div> 2304 380 16% 0.02K 16<span class="" style="white-space:pre"> </span>144 64K anon_vma</div><div> 2256 1852 82% 0.08K 47<span class="" style="white-space:pre"> </span> 48<span class="" style="white-space:pre"> </span> 188K selinux_inode_security</div>
<div> 2121 1943 91% 0.55K 303 7<span class="" style="white-space:pre"> </span> 1212K inode_cache</div><div> 1776 1463 82% 0.50K 222 8<span class="" style="white-space:pre"> </span> 888K size-512</div>
<div> 1710 705 41% 0.25K 114<span class="" style="white-space:pre"> </span> 15<span class="" style="white-space:pre"> </span> 456K filp</div><div> 1698 1642 96% 0.58K 283 6<span class="" style="white-space:pre"> </span> 1132K proc_inode_cache</div>
<div> 1632 1606 98% 2.00K 816 2<span class="" style="white-space:pre"> </span> 3264K size-2048</div><div> 1590 1147 72% 0.25K 106<span class="" style="white-space:pre"> </span> 15<span class="" style="white-space:pre"> </span> 424K skbuff_head_cache</div>
<div> 1584 324 20% 0.02K 11<span class="" style="white-space:pre"> </span>144 44K numa_policy</div><div> 1180 359 30% 0.06K 20<span class="" style="white-space:pre"> </span> 59 80K delayacct_cache</div>
<div> 1140 1101 96% 0.74K 228 5<span class="" style="white-space:pre"> </span> 912K ext3_inode_cache</div><div> 1080 1049 97% 0.09K 27<span class="" style="white-space:pre"> </span> 40<span class="" style="white-space:pre"> </span> 108K drbd_ee</div>
<div> 1054 1024 97% 0.11K 31<span class="" style="white-space:pre"> </span> 34<span class="" style="white-space:pre"> </span> 124K drbd_req</div><div> 1010 339 33% 0.02K<span class="" style="white-space:pre"> </span> 5<span class="" style="white-space:pre"> </span>202 20K biovec-1</div>
<div> 1008 888 88% 0.03K<span class="" style="white-space:pre"> </span> 9<span class="" style="white-space:pre"> </span>112 36K Acpi-Namespace</div><div> 944 335 35% 0.06K 16<span class="" style="white-space:pre"> </span> 59 64K pid</div>
<div> 650 514 79% 0.75K 130 5<span class="" style="white-space:pre"> </span> 520K shmem_inode_cache</div><div> 630 542 86% 0.12K 21<span class="" style="white-space:pre"> </span> 30 84K bio</div>
<div> 558 353 63% 0.81K 62 9<span class="" style="white-space:pre"> </span> 496K signal_cache</div><div> 496 496 100% 4.00K 496 1<span class="" style="white-space:pre"> </span> 1984K size-4096</div>
<div> 410 351 85% 1.84K 205 2<span class="" style="white-space:pre"> </span> 820K task_struct</div><div> 399 355 88% 2.06K 133 3<span class="" style="white-space:pre"> </span> 1064K sighand_cache</div>
<div> 354 54 15% 0.06K<span class="" style="white-space:pre"> </span> 6<span class="" style="white-space:pre"> </span> 59 24K fs_cache</div><div><br></div></div><div style>memtop:</div><div style><div>MemTotal: 65996216 kB</div>
<div>MemFree: 436188 kB</div><div>Buffers: 54272396 kB</div><div>Cached: 183784 kB</div><div>SwapCached: 0 kB</div><div>Active: 324660 kB</div><div>Inactive: 54143868 kB</div><div>
HighTotal: 0 kB</div><div>HighFree: 0 kB</div><div>LowTotal: 65996216 kB</div><div>LowFree: 436188 kB</div><div>SwapTotal: 8192504 kB</div><div>SwapFree: 8192388 kB</div><div>Dirty: 0 kB</div>
<div>Writeback: 0 kB</div><div>AnonPages: 12320 kB</div><div>Mapped: 8312 kB</div><div>Slab: 10988324 kB</div><div>PageTables: 1584 kB</div><div>NFS_Unstable: 0 kB</div><div>
Bounce: 0 kB</div><div>CommitLimit: 41190612 kB</div><div>Committed_AS: 44772 kB</div><div>VmallocTotal: 34359738367 kB</div><div>VmallocUsed: 267000 kB</div><div>VmallocChunk: 34359471059 kB</div><div>
HugePages_Total: 0</div><div>HugePages_Free: 0</div><div>HugePages_Rsvd: 0</div><div>Hugepagesize: 2048 kB</div><div><br></div></div>-- <br>Lin Zhao<br>Project Lead of Messagebus<div><a href="https://wiki.groupondev.com/Message_Bus" target="_blank">https://wiki.groupondev.com/Message_Bus</a><br>
3101 Park Blvd, Palo Alto, CA 94306<br></div>
</div></div>