SysConfig.adoc #1

= System Configuration
Perforce Professional Services <p4consulting@perforce.com>
:revnumber: v2026.1
:revdate: 2026-09-09
:doctype: book
:icons: font
:toc:
:toclevels: 5
:sectnumlevels: 4
:xrefstyle: full
// Attribute for ifdef usage
:unix_doc: true

== Preface

A P4 Server runs best when the environment it operates in is tuned for optimal performance. This document describes Linux tuning optimizations for `sysctl` to help get the best performance from your infrastructure.

*Please Give Us Feedback*

Perforce welcomes feedback from our users. Please send any suggestions for improving this document or the SDP to p4consulting@perforce.com.

:sectnums:

== Linux Kernel Parameters - A Brief History

A common issue with many current Linux distributions is with special OS settings referred to as kernel settings, or parameters. Earlier Linux releases left many of those unset or set to very high values, leaving it to system administrators to determine if they needed to be reduced to prevent the server from having a particular resource overloaded ("swamped"). Eventually many Linux maintainers realized this resulted in poor user experiences, as systems would become sluggish, unresponsive or unstable due to those higher or unset values.

As a result, many of those new default values are set with the least powerful hardware in mind, typically desktop users trying to squeeze more usable life out of otherwise underpowered, older hardware hosts of all types. While this helped improve the experiences of a great percentage of Linux users, such settings leave many high powered enterprise class servers essentially stuck in "first gear", unable to use far greater resources available to the host. This often results in at least one resource trying to make up the difference, usually the processor. It's not uncommon to see one or more CPU cores on such systems hitting 100% usage, even as the system slows down.

Another common symptom is a sharp increase of queued and running Perforce commands and processes. Many administrators and users will believe the server has hung or crashed, when it is in reality running very slowly.

WARNING: Do not kill or force restart a Perforce server except at the direction of Perforce Support. Doing so could cause database corruption across multiple tables and require long downtimes to repair or recover damaged data, and some data loss is inevitable.

== Reviewing Kernel Parameters

Over many years several key kernel parameters, in combination with related P4 Server configurables, have been shown to improve overall server performance, many times dramatically. In this section we will review those settings and their potential impact.

NOTE: By no means is this an exhaustive list, and the values given are defaults that likely can be adjusted further. If some improvements are seen after they are set then Perforce administrators will likely want to consult with Perforce Consulting or other resources for Linux parameter expertise. Support can make suggestions, but ALL such changes should be tested with non-production server environments prior to implementation.

== Obtaining Current Settings

There are often well over 500-1000 kernel parameters available. To simplify looking through the list of available settings use this command to re-direct the current settings for the P4 server host to a file:

[source,shell]
----
sysctl -a > $(hostname -s).sysctl.out
----

If multiple machines are involved (commit/edge, proxies, brokers, and so on) it is key that all those machines are likewise checked.

== Using `parse_sysctl.pl`

Looking through `sysctl` output can be time consuming and can result in key settings being missed. A simple perl script, `parse_sysctl.pl`, can be used to "scan" sysctl output files. It produces a set of output that explains what the current settings are compared to the recommended changes.

The script is located at `/p4/sdp/Server/Unix/setup/parse_sysctl.pl`.

Run the script with the sysctl output file. For example:

[source,shell]
----
/p4/sdp/Server/Unix/setup/parse_sysctl.pl $(hostname -s).sysctl.out
----

Or, if you would like to capture the script output for later editing:

[source,shell]
----
/p4/sdp/Server/Unix/setup/parse_sysctl.pl company_sysctl.out > parsed_sysctl
----

NOTE: Many settings could be more appropriate to a given system. Always review the output created by this script to determine if any values need to be removed or adjusted, and always test prior to rolling out to a production system. The script provides a way to backup and restore your current settings, including a "reboot" fail safe built into the work flow.

== Kernel Parameters Explained

=== net.ipv4.tcp_syncookies

----
Recommended Value: 0
Default:            1
----

Helps in preventing SYN flood attack on the system. A value of 0 will disable it.

For more details, please see SYN Cookies and WSAECONNRESET Errors.

[EDITME-JOHN-HALBIG: Got a Portal/KB reference for this?]

=== net.ipv4.tcp_timestamps

----
Recommended Value: 1
Default:            0
----

A value of 1 enables timestamps for TCP.

Many distributions turn timestamps off to reduce performance spikes related to timestamp generation. 

[EDITME-JOHN-HALBIG: Why is it on? Dig up old notes]

=== net.ipv4.tcp_window_scaling

----
Recommended Value: 1
Default:            1
----

From Red Hat's white paper on network performance tuning (see <<_related_links,Related Links>>):

In the original TCP definition, the TCP segment header only contains an 8-bit value for the TCP Window Size, which is insufficient for the link speeds and memory capabilities of modern computing. The TCP Window Scaling extension was introduced to allow a larger TCP Receive Window. This is achieved by adding a scaling value to the TCP options which are added after the TCP header.

While this is already enabled on most modern distributions, it may still be set to 0 in some environments. Setting this value to one enables window scaling.

=== net.core.somaxconn

----
Recommended Value: 2048
Default:            128
Min/Max:            128/65535
----

This is the maximum number of unhandled client connections a socket can handle; most default settings have this as 128, which can cause resets and other connection failures, particularly for clients issuing many commands very quickly (such as automated build systems, among others).

NOTE: This setting has been set as high as 4096 in some installations.

=== net.core.netdev_max_backlog

----
Recommended Value: 5000
Default:            1000
Min/Max:            TBD
----

This is critical for servers under high loads, as this value controls incoming traffic between the network connection and actual processing by the TCP stack and onward.

NOTE: This value creates a queue for each CPU core. Traffic sent to this queue is based on the process ID, so it's possible for a single process to max out the queue and drop packets if other cores are idle.

NOTE: Many Linux network tuning guides recommend a value of 100K or more for this setting. This value may be increased for future versions if testing and user reports warrant it.

=== net.ipv4.tcp_max_syn_backlog

----
Recommended: 2048
Default:     TBD
Min/Max:     TBD/65,535 to 3,240,000
----

Note: Max value based on older kernels; newer kernels can set this to a higher value, but it will be silently capped at the somaxconn value.

This value sets many half-open connections for which the client has not yet sent an ACK response can be kept in the network queue. More connections than this would be dropped.

Note: This value may still be TOO LOW; Many distributions set this to about 2048 now, and there are arguments for raising this value to 4096 or as high as  3,240,000.

There is a trick to calculate this; set `net.ipv4.tcp_syncookies` to 1 (enable cookies) and use this command to look for how many cookies your system is sending under high loads:

[source,shell]
----
netstat -s | grep "SYNs to LISTEN"
----

The 'SYNs to LISTEN' message is an indication that your server could be dropping packets due to this value being too low.

Note: This value is also limited by somaxconn and netdev_max_backlog settings in terms of kernel configurables, and ultimately limited by the maximum number of file handles the OS can handle, as two handles are opened for each network connection. For example:

[source,shell]
----
cat /proc/sys/fs/file-nr
8160    0       3270712
----

Here the OS is reporting 8,160 file handles out of over 3 million available.

=== net.ipv4.ip_local_port_range

----
Recommended: 10000 65535
Default:     32768 61000
Min/Max:     TBD
----

Two values are set to specify a range of ports available to client/server applications for this host. The default value provides 28,232 ports, but servers with a lot of connections being held open by custom clients, large numbers of pull threads, and high overall loads may exhaust this number. The new range nearly doubles the number of available ports to 55,535 ports.

=== net.ipv4.tcp_fin_timeout

----
Recommended: 30
Default:     60
Min/Max:     0/2147483
----

How many seconds to wait for a final FIN packet before the socket is forcibly closed. A FIN packet is usually sent from server or client to terminate a connection, after establishment of TCP 3-way handshake and successful transfer of data. Set at too high a value and the system can run out of ports, file handles or memory. Too low a value could leak delayed packets.

Note: Some older distributions set this as high as 180 seconds. This may be a value that can be tweaked lower, but not by very much.

=== net.ipv4.tcp_keepalive_time

----
Recommended: 890
Default:     7200
Min/Max:     TBD
----

The interval between the last data packet sent (simple ACKs are not considered data) and the first keepalive probe to determine if the connection should be held open. Once set, this value is not used again.

Note: This value works closely with the tcp_keepalive_intvl and the tcp_keepalive_probes values (below):

. The `tcp_keepalive_time` probe time limit is reached, and the system sends the first ACK packet.
. Regardless of whether data is received, a new probe is sent every `tcp_keepalive_intvl` seconds to confirm the connection is active.
. After the number of failures (as set by `tcp_keepalive_probes`) the connection is declared "dead" and reported to the application layer.

So the adjusted values for those settings substantially reduce the wait until the first keepalive ack from two hours to 15 minutes, shorten the time between acks from 75 to 30 seconds, but increase the number of probes from 9 to 20. This should prevent most completely dead connections from tying up server resources, while providing connections over poorer network conditions a chance to prove that, while mostly dead, they are still partly alive.

=== net.ipv4.tcp_keepalive_intvl

----
Recommended: 30
Default:     75
Min/Max:     TBD
----

The interval between sequential keepalive probes, regardless of what data was exchanged after keepalive probes are sent after the keepalive interval has been reached.

Note: See net.ipv4.tcp_keepalive_time, above.

=== net.ipv4.tcp_keepalive_probes

----
Recommended: 20
Default:     9
Min/Max:     TBD
----

The number of unacknowledged probes to send before considering the connection dead and notifying the application layer to drop the connection.

Note: See net.ipv4.tcp_keepalive_time, above.

=== net.core.netdev_budget

----
Recommended: 600
Default:     300
Min/Max:     TBD
----

If the software interrupt doesn’t process packets for a long time, it may cause the NIC buffer to overflow and, hence, can cause packet loss. netdev_budget shows the default value of the time period for which software interrupts should run.

Most higher end network cards (10Gbps or higher) need this to be set to 600; currently there is no benefit to increasing this value further.

=== net.core.rmem_max

----
Recommended: 16777216
Default:     212992
Min/Max:     TBD
----

Sets the maximum size for the read buffer for network connections, in bytes.

Note: This is connected to wmem_max, tcp_rmem, and tcp_wmem values, detailed below.

The chief risk of increasing this value is using too much memory; however, the default value (about 200K) is typically much too small for higher end systems to function well. This changes the maximum value to 16MB in size, which is only used as needed; the starting values for the read and write buffers (set by tcp_rmem and tcp_wmem) prevent this from using too much memory. This can be increased, but there is a point of diminishing returns; only benchmarking the specific server and loads will confirm that the newer values help or do nothing to improve performance.

=== net.core.wmem_max

----
Recommended: 16777216
Default:     212992
Min/Max:     TBD
----

Sets the maximum size for the write (send) buffer for network connections, in bytes.

Note: See net.core.rmem_max for more details.

=== net.ipv4.tcp_rmem

----
Recommended: 4096 87380 16777216
Default:     TBD
Min/Max:     1/TBD
----

Contains three values that represent the minimum, default and maximum size of the TCP socket receive buffer. Values are in 4K pages, so a setting of 1 would be 4096 bytes.

Note: See net.core.rmem_max for more details.

=== net.ipv4.tcp_wmem

----
Recommended: 4096 65536 16777216
Default:     TBD
Min/Max:     1/TBD
----

Contains three values that represent the minimum, default and maximum size of the TCP socket write (send) buffer. Values are in 4K pages, so a setting of 1 would be 4096 bytes.

Note: See net.core.rmem_max for more details.


=== net.core.optmem_max

----
Recommended: 16777216
Default:     20480
Min/Max:     TBD
----

The kernel option that affects the memory allocated to the list maintained by the kernel that contains "extra" packet information like SCM_RIGHTS or IP_TTL. Increasing this option allows the kernel to allocate more memory as needed for more control messages that need to be sent for each socket connected (including IPC sockets/pipes).

=== net.ipv4.tcp_mem

----
Recommended: 1528512 2038016 8388608
Default:     TBD
Min/Max:     TBD
----

The tcp_mem variable defines how the TCP stack should behave when it comes to memory usage.

The first value specified in the tcp_mem controls the lower threshold; below this point, the TCP stack does not attempt to control memory usage by network sockets. The second value tells the kernel at which point to start controlling network socket memory usage. The final value is the maximum amount of memory to be used for networking; if this value is reached, TCP streams and packets start getting dropped until a lower memory usage is achieved. This value includes all TCP sockets currently in use.

=== net.ipv4.tcp_no_metrics_save

----
Recommended: 1
Default:     0
----

By default, TCP saves various connection metrics in the route cache when the connection closes, so that connections established in the near future can use these to set initial conditions. Usually, this increases overall performance, but may sometimes cause performance degradation. If set, TCP will not cache metrics on closing connections.

=== net.ipv4.tcp_sack

----
Recommended: 0
Default:     1
----

Enable select acknowledgments (SACKs).


=== net.ipv4.conf.all.accept_redirects

----
Recommended: 0
Default:     1
----

Accept ICMP redirect messages. accept_redirects for the interface will be enabled if:

* Both conf/\{all,interface}/accept_redirects are TRUE in the case forwarding for the interface is enabled or;
* At least one of conf/\{all,interface}/accept_redirects is TRUE in the case forwarding for the interface is disabled accept_redirects for the interface will be disabled otherwise default TRUE (host) FALSE (router).

Note: a per interface setting (where “interface” is the name of your network interface); “all” is a special interface: changes the settings for all interfaces.

=== net.ipv4.conf.all.rp_filter

----
Recommended: 1
Default:     0
----

0 - No source validation.

1 - Strict mode as defined in RFC3704 Strict Reverse Path Each incoming packet is tested against the FIB and if the interface is not the best reverse path the packet check will fail. By default failed packets are discarded.

2 - Loose mode as defined in RFC3704 Loose Reverse Path Each incoming packet’s source address is also tested against the FIB and if the source address is not reachable via any interface the packet check will fail.

Current recommended practice in RFC3704 is to enable strict mode to prevent IP spoofing from DDoS attacks. If using asymmetric routing or other complicated routing, then loose mode is recommended.

The max value from conf/{all,interface}/rp_filter is used when doing source validation on the {interface}.

Note: Some distributions enable it in startup scripts.

Note: per interface setting (where “interface” is the name of your network interface); “all” is a special interface: changes the settings for all interfaces.

=== net.ipv4.conf.all.secure_redirects

----
Recommended: 0
Default:     1
----

Accept ICMP redirect messages only to gateways listed in the interface’s current gateway list. Even if disabled, RFC1122 redirect rules still apply. Overridden by shared_media. secure_redirects for the interface will be enabled if at least one of conf/{all,interface}/secure_redirects is set to TRUE, it will be disabled otherwise.

Note: per interface setting (where “interface” is the name of your network interface); “all” is a special interface: changes the settings for all interfaces.

=== net.ipv4.conf.all.send_redirects

----
Recommended: 0
Default:     1
----

Send redirects, if router. send_redirects for the interface will be enabled if at least one of conf/{all,interface}/send_redirects is set to TRUE, it will be disabled otherwise.

Note: per interface setting (where “interface” is the name of your network interface); “all” is a special interface: changes the settings for all interfaces.

=== net.ipv4.conf.all.accept_source_route

----
Recommended: 0
Default:     1
----

Accept packets with SRR option. conf/all/accept_source_route must also be set to TRUE to accept packets with SRR option on the interface.

=== net.ipv4.icmp_echo_ignore_broadcasts

----
Recommended: 1
Default:     1
----

If set non-zero, then the kernel will ignore all ICMP ECHO and TIMESTAMP requests sent to it via broadcast/multicast.

=== net.ipv4.ip_forward

----
Recommended: 0
Default:     1
----

Forward Packets between interfaces.

=== Rarely Used Settings

While not commonly used, those settings have been changed for customers in the past; currently set aside as there is some question as to their general suitability. Do not use those settings unless you have a solid basis for doing so, and test thoroughly for any undesirable performance or resource usage impacts after changing.

Note: Those settings may be moved to another knowledge base article to avoid article size limitations.

=== kernel.nmi_watchdog

----
Recommended: 0
Default:     1
----

An NMI watchdog is a watchdog which will generate a non maskable interrupt, i.e. the interrupt handler will get executed no matter what the CPU state is.

This is very useful in scenarios where you are getting unexplained system freeze scenarios, as the NMI watchdog interrupt handler will simply kill whatever process happens to be freezing the CPU at the moment. This way, your CPU gets freed up AND you get a detailed stack trace of why your CPU got frozen up in the first place.


=== kernel.soft_watchdog

----
Recommended: 1
Default:     1
----

The soft lockup detector monitors CPUs for threads that are hogging the CPUs without rescheduling voluntarily, and thus prevent the ‘watchdog/N’ threads from running. The mechanism depends on the CPUs ability to respond to timer interrupts which are needed for the ‘watchdog/N’ threads to be woken up by the watchdog timer function, otherwise the NMI watchdog - if enabled - can detect a hard lockup condition.


=== kernel.watchdog_thresh

----
Recommended: 180
Default:     10
Min/Max:     1/TBD
----

If a CPU fails to respond for the number of seconds the value is set to, throw a kernel panic. As a very busy system may see a delay in responding to an interrupt as a possible crash, increasing this value may reduce kernel panics at the risk of failing to detect a "real" panic until 3 minutes have passed.

=== kernel.softlockup_all_cpu_backtrace

----
Recommended: 1
Default:     0
----

This value controls the soft lockup detector thread’s behavior when a soft lockup condition is detected as to whether or not to gather further debug information. If enabled, each cpu will be issued an NMI and instructed to capture stack trace.

This feature is only applicable for architectures which support NMI.


=== kernel.hardlockup_all_cpu_backtrace

----
Recommended: 0
Default:     1
----

This value controls the hard lockup detector behavior when a hard lockup condition is detected as to whether or not to gather further debug information. If enabled, arch-specific all-CPU stack dumping will be initiated.


=== kernel.sysrq

----
Recommended: 0
Default:     0
----

According to the Linux kernel documentation:

It is a 'magical' key combo you can hit which the kernel will respond to regardless of whatever else it is doing, even if the console is unresponsive.

As this functionality can be a security risk, this is often already off by default.

=== kernel.core_uses_pid

----
Recommended: 1
Default:     0
----

The default coredump filename is “core”. By setting core_uses_pid to 1, the coredump filename becomes core.PID. If core_pattern does not include “%p” (default does not) and core_uses_pid is set, then .PID will be appended to the filename.

=== kernel.msgmnb

----
Recommended: 65536
Default:     TBD
Min/Max:     65536/TBD
----

Defines the maximum size in bytes of a single message queue.

=== kernel.msgmax

----
Recommended: 65536
Default:     TBD
Min/Max:     TBD
----

Defines the maximum allowable size in bytes of any single message in a message queue. This value must not exceed the size of the queue (msgmnb).


=== kernel.shmmax

----
Recommended: 68719476736
Default:     TBD
Min/Max:     TBD
----

Defines the maximum shared memory segment allowed by the kernel, in bytes. For database workloads, generally a value no larger than 75% of the total memory on the system.

=== kernel.shmall

----
Recommended: 4294967296
Default:     TBD
Min/Max:     TBD
----

Defines the total number of shared memory pages that can be used on the system at one time. For database workloads this value is set to the result of shmmax divided by the hugepage size.

== Related Links

* https://access.redhat.com/sites/default/files/attachments/20150325_network_performance_tuning.pdf[Red Hat White Paper on Network Tuning]
* https://serverfault.com/questions/875035/sane-value-for-net-ipv4-tcp-max-syn-backlog-in-sysctl-conf[Sane value for net.ipv4.tcp_max_syn_backlog in sysctl.conf]
* https://medium.com/@pawilon/tuning-your-linux-kernel-and-haproxy-instance-for-high-loads-1a2105ea553e[Tuning your Linux kernel and HAProxy instance for high loads]
* https://access.redhat.com/solutions/41776[tcp_max_tw_buckets]
* https://wiki.mikejung.biz/Sysctl_tweaks#net.core.netdev_max_backlog[Sysctl tweaks]
* http://tldp.org/HOWTO/TCP-Keepalive-HOWTO/usingkeepalive.html[Using TCP keepalive under Linux]
* https://sysctl-explorer.net[Sysctl Explorer]