Tech Note 0032

Multigigabit

Tuning systems for multigigabit speeds

This article examines the systems integration challenges which must be addressed when moving data at speeds of two gigabits per second or faster.  See Tech Note 0023 for a more general discussion of system requirements.

MTP/IP can deliver data disk-to-disk at full 10 gigabit per second line speed across high latency WANs with real-time AES encryption and data integrity checks, all on off-the-shelf hardware.  This article explains the network, storage, and CPU configuration necessary to support that.

Data cannot be moved along a path faster than the slowest component of that path.  To move data at very high speeds, all the components of the path must be functioning at very high speeds.  This includes storage, CPU, operating system, drivers, switches, routers, links, and software.

It is not enough that each and every component be capable of high speed operation.  Each component must be coordinated with every other component to ensure that they work together in practice.

Storage

The most common performance limitation is storage.  Most data paths start and end in files.  Most storage devices rate their throughput according to the speed of their interface, but their real-world transfer speed is much, much slower.  Storage speed may vary greatly depending on how it is accessed.  Storage is usually fastest when accessed sequentially, in a single file, using large blocks.  Performance may be reduced dramatically when storage is accessed out-of-order, in multiple files at a time, or using small I/O operations.

Most consumer grade hard-drives are not capable of sustaining multigigabit throughput.  Achieving high throughput, particularly for write operations, requires specialized storage hardware.  See the Storage section of Tech Note 0023 for general storage guidance.

Network Devices

A network path consists of many devices, ranging from the Network Interface Cards of the hosts, to switches, routers, firewalls, and wide area links.  A very common mistake in multigigabit networking is to focus on only one of these components.  For example, a wide area link might be 10 gigabits per second, but the NIC of a host might be only 1 gigabit per second.  Such a path would be limited to at most 1 gigabit per second, even though the "network" is 10 gigabits per second.

In addition to the multigigabit specific considerations above, all the usual network constraints discussed in Tech Note 0009 also apply.  Remember that a flaw in one device may cause unexpected behavior in a seemingly unrelated device.  For example, high latency in a storage device at one end of the path can be triggered by a misconfigured MTU at the other end of the path.

Hosts

Multigigabit throughput requires that each host system process hundreds of thousands of network datagrams and store or retrieve billions of bytes per second.  Most operating systems require special configuration for this level of performance.

Software

MTP/IP will attempt to adapt its behavior to match the requirements of the network, hosts, and storage.  But for optimal performance at multigigabit speeds, it may be necessary to provide it with some guidance.

Other software sharing an MTP/IP host may require adjustment so as to minimize its impact on network and storage performance.  In general, only software which is essential to the network task should be installed.

Windows is especially vulnerable to having extra software installed or running.  For example, an RDP session in which a user simply clicks the desktop once per second can cause a 25% drop in network throughput.  All user logins should be logged out and all RDP sessions disconnected to ensure maximum performance.

Summary

Multigigabit performance is difficult because it pushes current hardware to or beyond its limits.  As hardware capabilities improve, and components capable of operating at these speeds become more common, many of the considerations above will become less significant and the challenge of high performance will move on to higher orders of speed.

Tech Note History

Jan242022Minor updates
Jul272021Updated RAM advice
Apr062021Updated MTU advice
May172016Minor updates
Dec082014Updated Hosts section
Apr302014First Post