Recommended Linux tuning parameters for use with RV.
Article ID: KB0091767
Updated On:
| Products | Versions |
|---|---|
| TIBCO Rendezvous | - |
| Not Applicable | - |
Description
Resolution:
Description:
= = = = = = =
Recommended Linux tuning parameters for use with RV.
Environment:
= = = = = = =
Linux
Resolution:
= = = = = =
1. We have seen instances on linux boxes where there is dataloss and high retransmission if an internal queue feeding the NIC device is overrun. The problem could be due to overrun of a device's transmission queue.
High value is recommended for server connected over the high-speed Internet connections that perform large data transfers. The transmission queue depth can be changed by the use of 'ifconfig'. Setting it to 1000 can eliminate the drops.
If the data transfer is at Gig rates, a txqueuelen of 5000 - 10000 is recommended.
The command is:
ifconfig eth0 txqueuelen <length>
where <length> is replaced by the number of slots you want to reserve for the device's FIFO transmission queue.
Checking for packet loss using netstat -s and ifconfig are helpful.
However, if you want to see the data loss from tx_queue, you may need to modify the queue that is used. Since the qdisc stats are not available for txqueue, you have to add the qdisc with a FIFO queue of the same length (package iproute). This will allow you to see the dropped packets where previously the packets could be silently dropped.
a. Add the FIFO queue
$ ifconfig eth0 txqueuelen 1000
$ tc qdisc add dev eth0 root pfifo limit 1000
b. view resulting stats
$ tc -s -d qdisc show dev eth0
c. remove the FIFO queue
$ tc qdisc del dev eth0 root
To see the backlog queue you can use 'cat /proc/net/softnet_stat'.
2. We also have resolved Linux dataloss problems by suggesting the following tuning parameters:
Included are the values we suggested. You should discuss the tuning changes with your Linux vendor to ensure they are appropriate to your environment.
sysctl -a | grep mem
This will display your current buffer settings. Save these as you may want to roll-back these changes.
sysctl -w net.core.rmem_max= 16777216
This sets the max OS receive buffer size for all types of connections.
sysctl -w net.core.wmem_max= 16777216
This sets the max OS send buffer size for all types of connections.
net.core.rmem_default = 16777216
This sets the default OS receive buffer size for all types of connections.
net.core.wmem_default = 16777216
This sets the default OS send buffer size for all types of connections.
net.ipv4.tcp_mem = 16777216 16777216 16777216
TCP Autotuning setting. "The first value tells the kernel the minimum receive buffer for each TCP connection, and this buffer is always allocated to a TCP socket, even under high pressure on the system. ... The second value specified tells the kernel the default receive buffer allocated for each TCP socket. This value overrides the /proc/sys/net/core/rmem_default value used by other protocols. ... The third and last value specified in this variable specifies the maximum receive buffer that can be allocated for a TCP socket."
net.ipv4.tcp_wmem = 4096 65536 16777216
TCP Autotuning setting. "This variable takes 3 different values which holds information on how much TCP sendbuffer memory space each TCP socket has to use. Every TCP socket has this much buffer space to use before the buffer is filled up. Each of the three values is used under different conditions. ... The first value in this variable tells the minimum TCP send buffer space available for a single TCP socket. ... The second value in the variable tells us the default buffer space allowed for a single TCP socket to use. ... The third value tells the kernel the maximum TCP send buffer space."
Linux 2.4 Allows tcp window size to not stay reduced for new connections
net.ipv4.route.flush=1
Linux 2.6 don't cache ssthresh from previous connection which allows tcp window size to not stay reduced for new connections
net.ipv4.tcp_no_metrics_save = 1
net.ipv4.tcp_moderate_rcvbuf = 1
increase the receive packet queue for all Linux versions
net.core.netdev_max_backlog = 30000
3. We have customers have resolved high Rendezvous re-transmission rates on Linux by binding interrupts from each NIC driver to a single CPU. (Different NICs can be bound to different CPUs.)
You must know which interrupt is associated with each NIC in order to do this. You can find that information as well as seeing whether the interrupt is already bound to a CPU by looking at /proc/interrupts. Here is an example of that text file:
244:/proc>cat /proc/interrupts
CPU0 CPU1 CPU2 CPU3
0: 32985245 32977945 32977945 32977945 IO-APIC-edge timer
1: 31 2 2 1 IO-APIC-edge keyboard
2: 0 0 0 0 XT-PIC cascade
8: 1 0 0 0 IO-APIC-edge rtc
10: 0 0 0 0 IO-APIC-level usb-ohci
12: 30 179 107 0 IO-APIC-edge PS/2 Mouse
14: 3401108 3761 1808187 5187851 IO-APIC-edge ide0
16: 106239 0 0 0 IO-APIC-level eth1
29: 6 6883691 0 0 IO-APIC-level eth0
30: 423261 1112178 744897 24040 IO-APIC-level aic7xxx
31: 33 0 0 0 IO-APIC-level aic7xxx
NMI: 0 0 0 0
LOC: 131924541 131924540 131924539 131924539
ERR: 0
MIS: 0
In the example above, eth0 uses IRQ 29, and eth1 uses 16. By looking at the count of interrupts handled by each CPU, we see that all or almost all interrupts for each NIC are handled by the same CPU. That tells us that this machine is configured properly. (The machine only has two processors; the other two were invented by hyperthreading.)
If the counts are split across CPUs, you need to bind the IRQ(s) to a CPU(s).
For example, to bind both NIC IRQs 16 and 29 to processor 0, execute the following lines by appending them to /etc/rc.d/rc.local and rebooting:
echo "1" > /proc/irq/16/smp_affinity
echo "1" > /proc/irq/29/smp_affinity
Details of the CPU mapping values can be found on the Linux machine in /usr/src/<linux-version>/Documentation/filesystems/proc.txt
Please contact your Linux vendor for your tuning plans. Many RV performance issues have been resolved by applying the latest Linux patches.
Description:
= = = = = = =
Recommended Linux tuning parameters for use with RV.
Environment:
= = = = = = =
Linux
Resolution:
= = = = = =
1. We have seen instances on linux boxes where there is dataloss and high retransmission if an internal queue feeding the NIC device is overrun. The problem could be due to overrun of a device's transmission queue.
High value is recommended for server connected over the high-speed Internet connections that perform large data transfers. The transmission queue depth can be changed by the use of 'ifconfig'. Setting it to 1000 can eliminate the drops.
If the data transfer is at Gig rates, a txqueuelen of 5000 - 10000 is recommended.
The command is:
ifconfig eth0 txqueuelen <length>
where <length> is replaced by the number of slots you want to reserve for the device's FIFO transmission queue.
Checking for packet loss using netstat -s and ifconfig are helpful.
However, if you want to see the data loss from tx_queue, you may need to modify the queue that is used. Since the qdisc stats are not available for txqueue, you have to add the qdisc with a FIFO queue of the same length (package iproute). This will allow you to see the dropped packets where previously the packets could be silently dropped.
a. Add the FIFO queue
$ ifconfig eth0 txqueuelen 1000
$ tc qdisc add dev eth0 root pfifo limit 1000
b. view resulting stats
$ tc -s -d qdisc show dev eth0
c. remove the FIFO queue
$ tc qdisc del dev eth0 root
To see the backlog queue you can use 'cat /proc/net/softnet_stat'.
2. We also have resolved Linux dataloss problems by suggesting the following tuning parameters:
Included are the values we suggested. You should discuss the tuning changes with your Linux vendor to ensure they are appropriate to your environment.
sysctl -a | grep mem
This will display your current buffer settings. Save these as you may want to roll-back these changes.
sysctl -w net.core.rmem_max= 16777216
This sets the max OS receive buffer size for all types of connections.
sysctl -w net.core.wmem_max= 16777216
This sets the max OS send buffer size for all types of connections.
net.core.rmem_default = 16777216
This sets the default OS receive buffer size for all types of connections.
net.core.wmem_default = 16777216
This sets the default OS send buffer size for all types of connections.
net.ipv4.tcp_mem = 16777216 16777216 16777216
TCP Autotuning setting. "The first value tells the kernel the minimum receive buffer for each TCP connection, and this buffer is always allocated to a TCP socket, even under high pressure on the system. ... The second value specified tells the kernel the default receive buffer allocated for each TCP socket. This value overrides the /proc/sys/net/core/rmem_default value used by other protocols. ... The third and last value specified in this variable specifies the maximum receive buffer that can be allocated for a TCP socket."
net.ipv4.tcp_wmem = 4096 65536 16777216
TCP Autotuning setting. "This variable takes 3 different values which holds information on how much TCP sendbuffer memory space each TCP socket has to use. Every TCP socket has this much buffer space to use before the buffer is filled up. Each of the three values is used under different conditions. ... The first value in this variable tells the minimum TCP send buffer space available for a single TCP socket. ... The second value in the variable tells us the default buffer space allowed for a single TCP socket to use. ... The third value tells the kernel the maximum TCP send buffer space."
Linux 2.4 Allows tcp window size to not stay reduced for new connections
net.ipv4.route.flush=1
Linux 2.6 don't cache ssthresh from previous connection which allows tcp window size to not stay reduced for new connections
net.ipv4.tcp_no_metrics_save = 1
net.ipv4.tcp_moderate_rcvbuf = 1
increase the receive packet queue for all Linux versions
net.core.netdev_max_backlog = 30000
3. We have customers have resolved high Rendezvous re-transmission rates on Linux by binding interrupts from each NIC driver to a single CPU. (Different NICs can be bound to different CPUs.)
You must know which interrupt is associated with each NIC in order to do this. You can find that information as well as seeing whether the interrupt is already bound to a CPU by looking at /proc/interrupts. Here is an example of that text file:
244:/proc>cat /proc/interrupts
CPU0 CPU1 CPU2 CPU3
0: 32985245 32977945 32977945 32977945 IO-APIC-edge timer
1: 31 2 2 1 IO-APIC-edge keyboard
2: 0 0 0 0 XT-PIC cascade
8: 1 0 0 0 IO-APIC-edge rtc
10: 0 0 0 0 IO-APIC-level usb-ohci
12: 30 179 107 0 IO-APIC-edge PS/2 Mouse
14: 3401108 3761 1808187 5187851 IO-APIC-edge ide0
16: 106239 0 0 0 IO-APIC-level eth1
29: 6 6883691 0 0 IO-APIC-level eth0
30: 423261 1112178 744897 24040 IO-APIC-level aic7xxx
31: 33 0 0 0 IO-APIC-level aic7xxx
NMI: 0 0 0 0
LOC: 131924541 131924540 131924539 131924539
ERR: 0
MIS: 0
In the example above, eth0 uses IRQ 29, and eth1 uses 16. By looking at the count of interrupts handled by each CPU, we see that all or almost all interrupts for each NIC are handled by the same CPU. That tells us that this machine is configured properly. (The machine only has two processors; the other two were invented by hyperthreading.)
If the counts are split across CPUs, you need to bind the IRQ(s) to a CPU(s).
For example, to bind both NIC IRQs 16 and 29 to processor 0, execute the following lines by appending them to /etc/rc.d/rc.local and rebooting:
echo "1" > /proc/irq/16/smp_affinity
echo "1" > /proc/irq/29/smp_affinity
Details of the CPU mapping values can be found on the Linux machine in /usr/src/<linux-version>/Documentation/filesystems/proc.txt
Please contact your Linux vendor for your tuning plans. Many RV performance issues have been resolved by applying the latest Linux patches.
Issue/Introduction
Recommended Linux tuning parameters for use with RV.
Feedback
Yes
No
Powered by
