[DRBD-user] Page allocation failure (IPOIB, Infiniband, connected mode)

[Christian Balzer]

Hello,

This is basically a repeat of:
http://lists.linbit.com/pipermail/drbd-user/2011-August/016758.html

32GB RAM, Debian Squeeze, 3.2 (debian backport) kernel, 8.3.12 DRBD,
IPOIB in connected mode with a 64k MTU. Just 2 DRBD resources.

After encountering this for the first time (never showed up in two weeks
of stress testing, which only goes to prove that real life just can't be
simulated) I found the above article and changed the following sysctls:

vm/min_free_kbytes = 262144
net.core.rmem_max = 16777216 
net.core.wmem_max = 16777216 
net.ipv4.tcp_rmem = 65536 87380 16777216
net.ipv4.tcp_wmem = 65536 65536 16777216

net/ipv4/tcp_mem is by default at "775464  1033954 1550928" which should be 
plentiful.

After the changes I disconnected and reconnected the resources, thus hopefully applying those changes to the actual links. 
And initially I couldn't reproduce the problem, but it showed up again last 
night.

Lars hinted at "atomic reserves" in his reply, which particular parameters
are we talking about here?

See this for the pertinent kernel log entry in my case:
http://pastebin.com/hTJuCYDX

Regards,

Christian
-- 
Christian Balzer        Network/Systems Engineer                
chibi at gol.com   	Global OnLine Japan/Fusion Communications
http://www.gol.com/