The MTP/IP networking technology that transports data for ExpeDat is designed to adapt to a wide variety of network circumstances. The default settings will allow you to use the full available resources of your network while still being fair to other traffic.
If you are not getting the performance you expect, read this section carefully or contact technical support.
In addition to the information contained in this manual, detailed technical information about optimizing network performance can be found in the DEI Tech Notes. An index of the Tech Notes most relevant to performance can be found at the bottom of this page.
Most Wide Area Network paths involve hundreds of different components, any one of which may be limiting your available bandwidth. Here are a few of the more common types to consider:
- Slowest physical link in the path
- Disk I/O speed
- Network emulators (See Tech Note 0022.)
- Routers configured to meet Quality of Service goals
- CPU speed and cores
- Other applications using CPU or disk
- Bonded or multiplexed network links
- Faulty, non-standard, or obsolete routers and switches
- UDP buffers on Linux, FreeBSD, and Solaris (See Tech Note 0024.)
- MTU below 1500 bytes. (See Small MTU.)
ExpeDat cannot move data faster than the slowest of these capacities. Rate-limiting devices, especially firewalls, emulators, and routers, may not be treating all traffic equally! So if performance is not meeting your expectations, check your systems for the types of limitations listed above. DEI Tech Note 0003 offers detailed advice on testing and finding network limitations.
Filesystem bottlenecks can severely limit performance. This is particularly true for network attached storage. Below are some tips for optimizing storage performance:
- Use direct attached storage whenever possible.
- If you must use network attached storage (NAS), see Tech Note 0029 for performance tuning tips.
- RAIDs are not always faster. See Tech Note 0018.
- Linux systems may suffer from excessive caching. See Tech Note 0035 for adjusting cache sizes.
- Hard drive performance drops exponentially with the number of users:
Two parallel high-speed transfers will add up to less than one!
- For Windows, NTFS formatted filesystems may provide much better write performance than FAT.
- Hard drive performance drops when the disk is nearly full or if a Windows system has not been defragmented in some time. Use a drive with plenty of free space and defragment Windows systems.
- When using NFS filesystems, make sure you have an adequate number of biod and nfsd processes on the NFS clients and servers. To avoid potentially critical NFS bugs, use version 4 or later with up-to-date patches.
- For the most consistent performance with EC2 Elastic Block Storage (EBS), use an EBS-Optimized volume with the desired bitrate. See Tech Note 0025 for more about installing on EC2.
- Inline decompression can create a much higher load on the filesystem than it does on the network. See Tech Note 0014 for more about compression performance.
- Environmental factors, such as vibration in rack-mounted systems, can reduce drive performance far below rated specifications. Be sure your mount points are well isolated and check all cables and connections.
- File transfers include the time it takes the OS to commit data to disk. If storage is slower than the network, there may be a long pause at the end of each transfer. See Write Confirmation for details.
Any time the total network throughput may be faster than about 100 megabits per second or you are using any form of network attached storage, you should investigate your filesystem's I/O capabilities. See Tech Note 0023 for further hardware recommendations.
A single modern CPU core can support 2 or 3 gigabits per second of throughput. For faster speeds, a good rule of thumb is to have one available CPU core for every 2 or 3 gigabits per second of desired throughput. For example, a ten gigabit per second path can be saturated with about 4 to 6 CPU cores, assuming the network and storage can keep up. When content encryption is enabled, the number of CPU cores needed roughly doubles.
If other CPU intensive applications are running on the same system, network performance may be reduced. This may happen even if there are idle CPU cores available. Applications which filter or monitor network traffic may substantially reduce network performance. For example, running Windows Performance Monitor may reduce network throughput below one gigabit per second. For the best multigigabit performance, avoid running anti-virus, firewall, or monitoring software on the same hardware as ExpeDat.
Virtualized CPUs perform well up to about one gigabit per second. Multigigabit speeds will be impacted by the particular virtualization environment. For the best multigigabit performance, use real hardware or consult the virtualization vendor for network optimization guidance.
Some anti-virus software, including Windows Defender, may greatly reduce file transfer performance. Symptoms include slow scan times when sending large numbers of files, long delays near the end of a download, or poor network speeds accompanied by high CPU usage. If you are experiencing slow performance, particularly on a Windows system, you can test whether anti-virus software is the cause by temporarily disabling it to see if performance improves.
To avoid performance problems with anti-virus software, white list ExpeDat applications and file transfer directories. If applicable, configure the anti-virus software to delay scanning new files until after they are finished writing, disable heuristic analysis, and disable or make an exception for scanning network traffic.
Moving Data Even Faster
Normally MTP/IP makes an effort to allow other traffic on the network a fair share of available resources. If you want your ExpeDat data flows to achieve the absolute maximum speed, regardless of other users, you may do so by increasing the network "aggression" of MTP/IP.
Aggression affects relative performance compared to other data flows sharing the same network path, similar to prioritization. If there are no other data flows, or if performance is limited by hardware or policy, then increasing aggression may have little or no effect.
Aggression is a value between -3 and +5 which determines how hard MTP/IP presses the limits of the underlying network hardware. Default values are in the range of 0 to 2 (depending on your license configuration) and try to strike a balance between speed and fairness. It is very rare to need to set aggression higher than +2.
Setting aggression to +3, +4, or +5 may improve ExpeDat throughput, but it may also be disruptive to other users of the network. Such high aggression levels can sometimes result in worse performance if other bandwidth management devices are present.
Carefully read and understand the following paragraph before using aggression level 5 or MinDatagram settings.
If your network path is very fast (gigabit or more) and every device along that path supports Jumbo frames (MTU 9000) or Super Jumbo frames (MTU up to 65536), then you may be able to reduce I/O overhead and improve speed by advising MTP/IP to use larger datagrams. This can be accomplished by setting MinDatagram to 8192 for Jumbo frame networks or up to 61440 for Super Jumbo. MTP will automatically attempt to use 8192 byte payloads for an individual transfer exceeding 1 gigabits per second, unless MaxDatagram is set lower. Setting aggression to level 5 will also attempt to use larger datagrams, but setting MinDatagram is preferred. If any component of the path does not fully support large datagrams, their use may cause severe performance degradation or loss of connectivity.
Refer to each applications's documentation for instructions on setting aggression and Tech Note 0005 for more information about MTP/IP bandwidth management features.
Moving Data Slower
For shared network environments, you may wish to limit ExpeDat's impact on other users even more than the default settings.
The most flexible way to give priority to other users is to reduce the aggression level described above. Values below 2 will limit the resources MTP/IP is willing to use, leaving more resources for other traffic flows. Note that on networks with inherently high-latency or packet loss (such as satellite or packet radio), lower aggression may reduce performance even when there is no other traffic.
Aggression affects relative performance compared to other data flows sharing the same network path. If there are no other data flows and the network path is "clean", decreasing aggression may not have an obvious effect on performance. But it will still cause performance to slow down in the presence of other data flows.
You can also set specific bandwidth limits for ExpeDat. The MaxRate settings can be used to establish speed limits for both individual transactions and the server as a whole. If your data path has a fixed known capacity, you can set MaxRate to a value less than that to apportion the bandwidth between ExpeDat and other users. You can even combine this with increased aggression to throttle ExpeDat to exactly that amount of bandwidth.
If you have applications on your network which are sensitive to latency issues, such as voice or video over IP, you can set a specific latency ceiling for ExpeDat. The MaxRTT setting will cause MTP/IP to slow down whenever it observes latency above the given value.
In order to avoid stalls and other inefficiencies, MTP will always maintain at least one datagram on the network at any time. Thus, transactions cannot be throttled below 256 bytes per Round-Trip-Time. For example, a path with an RTT of 1ms has a minimum rate limit of about 2 megabits per second, while a path with 10ms latency has a minimum of about 200 kilobits per second. This affects both MaxRate and MaxRTT, primarily on LANs.
Refer to the documentation for servedat, ExpeDat Desktop, and movedat for details on how to set these values. Also see DEI Tech Note 0005 for more information about MTP/IP bandwidth management features.
Client & Server Settings
Performance can be regulated at either the client or server. Most server performance settings consist of a default value and a limit value. The default is used when the client does not specify a value, but the client may not go past the limit value.
Many server performance settings can be set according by the originating IP address of the client using the SiteOptions configuration variable. This allows administrators to regulate performance based on the client network.
The ExpeDat Desktop client also allows administrators embed settings within a copy of the application. This allows different performance profiles to be distributed to specific persons or systems. Each embedded settings can be locked to prevent the user from modifying it.
The following Tech Notes provide further guidance on performance tuning:
|0023||Recommended minimum hardware for high speed networking.|
|0005||Features for regulating MTP and ExpeDat bandwidth usage.|
|0029||Optimizing Network Attached Storage (NAS).|
|0003||Important tips on conducting accurate and meaningful performance tests.|
|0033||Instructions for capturing network diagnostics.|
|0021||Guidance for accurately measuring network Loss, Latency, and Speed.|
|0022||See this note if you are considering use of a network emulator for testing.|
|0009||Details about specific devices and factors which may degrade network performance.|
|0024||Operating system tuning for certain unix variants.|
|0018||Configuration requirements for RAIDs.|
|0014||Pros and cons of using inline compression.|
|0025||Using ExpeDat with Amazon Web Services EC2 instances.|