Posted: Fri Nov 26, 2010 7:55 am    Post subject: rmem_max problem in VC 6.1  

Tonight we had a BGP neighbour flapping several times, and out vyatta router collapsed. I found many errors about memory allocation:

Code:
Nov 26 03:26:36 milano1 zebra[2286]: netlink-cmd error: Cannot allocate memory, type=RTM_NEWROUTE(24), seq=18081945, pid=0
Nov 26 03:26:36 milano1 zebra[2286]: netlink-cmd error: Cannot allocate memory, type=RTM_NEWROUTE(24), seq=18081947, pid=0
Nov 26 03:26:36 milano1 zebra[2286]: netlink-cmd error: Cannot allocate memory, type=RTM_NEWROUTE(24), seq=18081949, pid=0
Nov 26 03:26:36 milano1 zebra[2286]: netlink-cmd error: Cannot allocate memory, type=RTM_NEWROUTE(24), seq=18081951, pid=0
....


even during normal operations we saw occasional messages like those.

we had a similar problem in the past with a plain zebra installation, and the solution was to raise the -s parameter when starting the zebra daemon

with a ps ax | grep zebra we noticed that VC 6.1 set a default -s 1048576, however, in the zebra documentation we see

Quote:


-s, --nl-bufsize netlink-buffer-size
Set netlink receive buffer size. There are cases where zebra
daemon can't handle flood of netlink messages from kernel. If
you ever see "recvmsg overrun" messages in zebra log, you are in
trouble.

Solution is to increase receive buffer of netlink socket. Note
that kernel doesn't allow to increase it over maximum value
defined in /proc/sys/net/core/rmem_max. If you want to do it,
you have to increase maximum before starting zebra.

Note that this affects Linux only.


it says that, if we set to N the netlink socket buffer size, the /proc/sys/net/core/rmem_max should be raised by the same amount. however, in our installation, we found that the vyatta installer set
/proc/sys/net/core/rmem_max=223232
/proc/sys/net/core/rmem_default=111616

we modified the sysctl.conf to

Code:
net.core.rmem_default = 1048576
net.core.rmem_max = 1048576
net.core.wmem_default = 1048576
net.core.wmem_max = 1048576


rebooted and the warnings were gone.

I think these modifications should go into the next release