Note: "permalinks" may not be as permanent as we would like,
direct links of old sources may well be a few messages off.
Hey all. I currently have a DRBD filesystem setup between 2 nodes both
running on CentOS 5.2 servers. Version information:
heartbeat-2.1.3-3.el5.centos
kmod-drbd82-8.2.6-1.2.6.18_92.1.10.el5
drbd82-8.2.6-1.el5.centos
kernel-2.6.18-92.1.10.el5
/proc/drbd
version: 8.2.6 (api:88/proto:86-88)
GIT-hash: 3e69822d3bb4920a8c1bfdf7d647169eba7d2eb4 build by
buildsvn at c5-i386-build, 2008-08-07 17:07:52
0: cs:Connected st:Primary/Secondary ds:UpToDate/UpToDate C r---
ns:409988212 nr:1220768 dw:411208980 dr:108310305 al:1353369 bm:1174
lo:0 pe:0 ua:0 ap:0 oos:0
The problem is that after some time I start to see memory usage on the
primary drbd node increase and never get free'd. This happens over the
course of a month. before the box (with a gig of ram) eventually starts
swapping. Under top i only see the heartbeat process using the most memory
which is only at 1.2%.
However when i run slabtop, I see a process 'rpc_tasks' consuming around
850M of the memory:
OBJS ACTIVE USE OBJ SIZE SLABS OBJ/SLAB CACHE SIZE NAME
3149025 3149025 100% 0.25K 209935 15 839740K rpc_tasks
93583 93530 99% 0.02K 461 203 1844K avtab_node
I've no idea what this task is, but I assume it has something to do with
NFS. The only way to free up the memory is to reboot the server.
Under /var/log/messages I see the following errors repeated hundreds of
times:
Dec 12 08:20:04 fs1 kernel: lockd: server 192.168.0.128 not responding,
timed out
Dec 12 14:52:16 fs1 kernel: lockd: server 192.168.0.128 not responding,
timed out
Dec 13 14:52:09 fs1 kernel: lockd: server 192.168.0.128 not responding,
timed out
The .128 host is an NFS client accessing data on the DRBD NFS share. There
are 2 other nodes here (.129 and .130) but neither of them have any
problems. And all of them have lockd / statd / etc running.
Both are mounted using NFS via UDP, not TCP.
Does anyone have any idea what the memory consumption error could be related
to? I can provide any additional information if necessary.
As a side note, I did see that Centos has a kernel update and an update to
the drbd kernel module, but since the rpc_tasks() (system call?) seems to be
chewing up the memory, I dont think upgrading drbd's module version will fix
it and I'm hesitant to upgrade to the newer kernel willy-nilly like.
I found this, but only found 2 fixes pertaining to NFS:
https://rhn.redhat.com/errata/RHSA-2008-1017.html
450335 - LTC41974-Pages of a memory mapped NFS file get corrupted.
469650 - [REG][5.2][NFSv4] Accessing the same file at the same time causes
NFSv4 open() call to stall forever on NFS4ERR_DELAY
Any help is greatly appreaciated.
--
View this message in context: http://www.nabble.com/rpc_tasks-slowly-consumes-memory-tp21083441p21083441.html
Sent from the DRBD - User mailing list archive at Nabble.com.