The txqueuelen is a value in the kernel on network interfaces that sets the transmit queue length. This value can be tuned for different work loads. In the case of modern networking the defaults can sometimes be changed to get better line speeds over ethernet. Most people will do this using a rc.local command to set it on the physical ethX devices like this.

vim /etc/rc.local

Add the following

/sbin/ip link set eth0 txqueuelen 2500

This is a perfectly reasonable way of doing it but what happens when network interfaces appear after boot and the name is unknown before hand? This is exactly what happens with Libvirt vnetX interfaces. Ideally we would be able to get Libvirt to tune these interfaces when it creates them, but that level of control is yet to be implemented BZ#809172. Libvirt >= 0.8.0 has some hooks which enable you to run commands at specific times in the lifecycle of a guest which may be good for this but on RHEL 5 Libvirt is version 0.6.0 so I needed a different solution.

In comes udev. Udev is the device manager for the Linux kernel and you can write rules for when devices are added and removed from the system. A common user visible use of udev is when a camera is plugged in, udev mounts it and informs the desktop environment of its availability. This allows the desktop environment to open a photo import application.

When you start a guest using Libvirt it creates vnetX devices to match the network interfaces seen inside the guest. These vnet devices are the ones I want to change the txqueuelen on. The default txqueuelen in our environment is 500 but with some experimentation we found we could increase the throughput of our network by a factor of 3 by changing it to 2500.

vnet6     Link encap:Ethernet  HWaddr 36:9A:FA:9C:38:0E
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:500
          RX bytes:0 (0.0 b)  TX bytes:0 (0.0 b)

The X in vnetX is a integer which is allocated on a ‘first free’ basis so we cannot tell what it will be prior to creation. If you already have a guest running you can look at the information available to udev by running the following command:

#> udevinfo -a -p /sys/class/net/vnet0/
  looking at device '/class/net/vnet0':
    KERNEL=="vnet0"
    SUBSYSTEM=="net"
    SYSFS{weight}=="0"
    SYSFS{tx_queue_len}=="2500"
    SYSFS{flags}=="0x1103"
    SYSFS{mtu}=="1500"
    SYSFS{operstate}=="unknown"
    SYSFS{dormant}=="0"
    SYSFS{carrier}=="1"
    SYSFS{broadcast}=="ff:ff:ff:ff:ff:ff"
    SYSFS{address}=="8e:77:cb:21:b8:79"
    SYSFS{link_mode}=="0"
    SYSFS{type}=="1"
    SYSFS{features}=="0x190049"
    SYSFS{ifindex}=="56"
    SYSFS{iflink}=="56"
    SYSFS{addr_len}=="6"

These values can be used to match the device in a rule. In this case we want all vnetX devices which we can do using the KERNEL value, this is the name allocated to the device by the kernel. The following will match any vnetX device that gets created

KERNEL=="vnet[0-9]*"

Then we want to run a command to change the txqueuelen when the kernel sees one of those devices change. The RUN value is a list of commands to be run after a device exists. These commands need to be very short lived as they pause udev while they are running.

RUN+="/sbin/ip link set %k txqueuelen 2500"

The final udev rule is:

KERNEL=="vnet[0-9]*", RUN+="/sbin/ip link set %k txqueuelen 2500"

This can be put into a file under /etc/udev/rules.d/ for example /etc/udev/rules.d/60-vnet.rules

Now when you start a guest os the vnet devices will all have a txqueuelen of 2500 (this value suits the specific workload we are running YMMV)

vnet6     Link encap:Ethernet  HWaddr A6:CE:E6:9F:39:FB
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:2500
          RX bytes:0 (0.0 b)  TX bytes:0 (0.0 b)

References: