Note: "permalinks" may not be as permanent as we would like,
direct links of old sources may well be a few messages off.
As a side note, I am unable to reboot the machine. A few minutes after running 'shutdown -r now' I see this on the console screen: ==================== BUG: soft lockup - CPU#0 stuck for 10s! [rpciod/0:3134] Pid: 3134, comm: rpciod/0 EIP: 0060:[<C060998E>] CPU: 0 EIP is at _spin_lock_bh+0xf/0x18 EFLAGS: 00000286 Tainted: G (2.6.18-92.1.10.el5 #1) EAX: f251b000 EBX: f38d7b58 ECX: d143ab84 EDX:90d91878 ESI: f38d7b58 EDI: 00000000 EBP: 00000246 DS: 007b ES: 007b CR0: 8005003b CR2: 00172fa0 CR3: 00726000 CR4: 000006d0 [<f99e89b9>] svc_wake_up+0xe/0x4b [sunrpc] [<f99e5c54>] rpc_exit_task+0x1b/0x58 [sunrpc] [<f99e5ed9>] __rpc_execute+0x7a/0x1f8 [sunrpc] [<c043322a>] run_workqueue+0x78/0xb5 [<f99e6057>] rpc_async_schedule+0x0/0x5 [sunrpc] [<c0433ade>] worker_thread+0xd9/0x10b [<c042027b>] defaulty_wake_function+0x0/0xc [<c0433a05>] worker_thread+0x0/0x10b [<c0435eed>] kthread+0xc0/0xeb [<c0435e2d>] kthread+0x0/0xeb [<c0405c3b>] kernel_thread_helper+0x7/0x10 ==================== Erik Lat wrote: > > Hey all. I currently have a DRBD filesystem setup between 2 nodes both > running on CentOS 5.2 servers. Version information: > > heartbeat-2.1.3-3.el5.centos > kmod-drbd82-8.2.6-1.2.6.18_92.1.10.el5 > drbd82-8.2.6-1.el5.centos > kernel-2.6.18-92.1.10.el5 > > /proc/drbd > version: 8.2.6 (api:88/proto:86-88) > GIT-hash: 3e69822d3bb4920a8c1bfdf7d647169eba7d2eb4 build by > buildsvn at c5-i386-build, 2008-08-07 17:07:52 > 0: cs:Connected st:Primary/Secondary ds:UpToDate/UpToDate C r--- > ns:409988212 nr:1220768 dw:411208980 dr:108310305 al:1353369 bm:1174 > lo:0 pe:0 ua:0 ap:0 oos:0 > > > The problem is that after some time I start to see memory usage on the > primary drbd node increase and never get free'd. This happens over the > course of a month. before the box (with a gig of ram) eventually starts > swapping. Under top i only see the heartbeat process using the most memory > which is only at 1.2%. > > However when i run slabtop, I see a process 'rpc_tasks' consuming around > 850M of the memory: > > OBJS ACTIVE USE OBJ SIZE SLABS OBJ/SLAB CACHE SIZE NAME > 3149025 3149025 100% 0.25K 209935 15 839740K rpc_tasks > 93583 93530 99% 0.02K 461 203 1844K avtab_node > > I've no idea what this task is, but I assume it has something to do with > NFS. The only way to free up the memory is to reboot the server. > > Under /var/log/messages I see the following errors repeated hundreds of > times: > > Dec 12 08:20:04 fs1 kernel: lockd: server 192.168.0.128 not responding, > timed out > Dec 12 14:52:16 fs1 kernel: lockd: server 192.168.0.128 not responding, > timed out > Dec 13 14:52:09 fs1 kernel: lockd: server 192.168.0.128 not responding, > timed out > > The .128 host is an NFS client accessing data on the DRBD NFS share. There > are 2 other nodes here (.129 and .130) but neither of them have any > problems. And all of them have lockd / statd / etc running. > Both are mounted using NFS via UDP, not TCP. > > Does anyone have any idea what the memory consumption error could be > related to? I can provide any additional information if necessary. > > As a side note, I did see that Centos has a kernel update and an update to > the drbd kernel module, but since the rpc_tasks() (system call?) seems to > be chewing up the memory, I dont think upgrading drbd's module version > will fix it and I'm hesitant to upgrade to the newer kernel willy-nilly > like. > > I found this, but only found 2 fixes pertaining to NFS: > https://rhn.redhat.com/errata/RHSA-2008-1017.html > 450335 - LTC41974-Pages of a memory mapped NFS file get corrupted. > 469650 - [REG][5.2][NFSv4] Accessing the same file at the same time causes > NFSv4 open() call to stall forever on NFS4ERR_DELAY > > Any help is greatly appreaciated. > > -- View this message in context: http://www.nabble.com/rpc_tasks-slowly-consumes-memory-tp21083441p21087081.html Sent from the DRBD - User mailing list archive at Nabble.com.