Tech Note 0003

Analyzing Network Performance

Best practices for testing network performance

The data path between any two computers involves dozens, sometimes thousands, of hardware and software devices.  Any one of these may have a substantial impact on performance.

At any given time there is likely to be one factor which most directly limits the maximum speed of a data flow.  Identifying the limiting factor for each data flow is vital to improving performance.

The limiting factor may change with time or in response to changes in other factors.  Changing network conditions may change the relative performance of MTP and TCP.  Test under a variety of conditions and learn all you can about your network environment.

Testing Tips

Tests should be conducted in the same real-world environments where you plan to deploy the software.  If that is not practical, great care must be taken to ensure that the test environment matches the production environment in equipment, network devices, and traffic patterns.  Even if the test conditions appear to be very similar to the intended use conditions, there may be many hidden factors which affect performance.

Keep Records
Record complete details of your testing so that you know exactly what you did and what effect it had.  This information is essential to assisting our engineers in diagnosing any performance questions you may have.
Avoid Emulators
Network emulators use statistical models that are designed around TCP and TCP-like network traffic.  While emulators can be useful, they must be carefully configured using statistics appropriate to the traffic being tested.  DEI strongly recommends testing MTP in real-world environments whenever possible.  If you must use an emulator, carefully read Tech Note 0022 for information on programming it with the best possible data.
Configure Firewalls, NAT, and VPN Devices
Firewalls, NAT gateways, and other such devices are designed to interfere with network traffic.  They may selectively block or degrade MTP/IP applications.  They are often configured to interfere with some traffic more than others.  For testing purposes, you must disable, or at least be aware of, any "bandwidth management", "packet shaping", "traffic shaping", "stateful security, "dynamic security", or other features which selectively slow down traffic.  Once a baseline is established, you can then re-enable these features one at a time to observe their effects.  See Tech Note 0002, "Configuring Firewalls" for instructions.
Disable TCP acceleration devices
You may already be using a hardware or software device to "accelerate" your network.  Such systems may be configurable to work with MTP, but for test purposes disable third-party acceleration devices.  You can re-enable them later once you have established a baseline of performance.
Measure Time Independently
Many FTP and HTTP clients (especially Internet Explorer) are very "optimistic" when reporting transfer speeds because they only report during bursts of data arrival and don't count the times when TCP is stalled.  For the most accurate results, measure the time it takes to transfer data using a stopwatch or a command line utility such as "time".
Test Different Times and Different Paths
TCP is very sensitive to the type of network being used and the patterns of traffic present.  Its performance will vary between different sites and throughout the day and week.  For the most complete picture of how MTP and TCP compare, try as wide a variety of tests sites and times as possible.

Factors Affecting MTP/IP Performance

The following factors affect MTP/IP directly, and will therefore affect all MTP/IP applications.

Maximum Path Speed
MTP cannot move data faster than the underlying network hardware.  When hardware is the limiting factor, MTP will not provide a throughput improvement, but will still provide reliability and transaction speed improvements.  Note that the maximum path speed is the actual maximum throughput of the slowest link between two computers.  Very often, hardware such as DSL lines and WiFi, or services such as Virtual Private Networks, are rated at speeds much higher than they actually deliver.  Determine your actual path speed before evaluating any performance options.
Network Devices
As described above, firewalls, emulators, routers, NATs, acceleration appliances, and other devices may selectively limit performance of some traffic flows and not others.  Be aware of any such devices and check their configurations carefully.  See Tech Note 0002 for configuration tips.
TCP Relative Speed
TCP's performance may vary substantially due to many factors, including the time of day.  If TCP is operating close to the Maximum Path Speed, then MTP may not offer faster throughput than TCP, because no transport protocol can be faster than the network itself.
Operating Systems
Different operating systems handle networking in different ways.  Windows is very different from unix systems.  The choice of system also affects how other factors such as CPU Load, Memory, Disk Speed, and Local Link Speed will affect each protocol and application.  See Tech Note 0024 for tunable parameters which affect some unix systems.
CPU Load
A high CPU load will affect both TCP and MTP performance.  Depending on the Operating System, type of software running, and many other factors, one protocol may be affected more than the other.  Using a faster processor and running fewer other applications may improve network performance for both protocols.
Third Party Traffic
MTP and TCP must compete with other users of the network.  As more people use a network path, performance for all users will decrease.  This is especially true if the Maximum Path Speed is the limiting factor.  TCP becomes very inefficient when third party traffic is high, causing TCP to consume large amounts of resources while providing poor performance.  MTP will show the largest relative performance improvement at times when third-party traffic is high.  Tech Note 0005 describes how MTP's impact on third-party traffic can be increased or decreased.
Idle Latency (Round Trip Time)
The time it takes for a very small amount of data to travel between computers and back again is called the Round Trip Time (RTT).  Rising RTT is indicative of heavy network use.  If RTT is high (over 500ms), TCP performance will often be poor and MTP performance relatively good.  If RTT is very low (under 10ms), TCP performance may be limited only by the Maximum Path Speed.  MTP can be configured to limit its effect on RTT.  See each application's documentation or contact DEI for details.

Factors Affecting ExpeDat Performance

The ExpeDat application is normally used to transfer files stored on the hard drive of one computer to the hard drive of another computer.  End-users may interact directly with ExpeDat, particularly the graphical client.  These functions may have an impact on data transport performance.  Note that ExpeDat will generally provide better performance than MTP Tunnel.

Disk Speed
ExpeDat cannot move data faster than the hard drives can read or write the data.  Hard drives become exponentially less efficient when there is more than one data flow at a time.  Two parallel high-speed transfers will add up to less than one!  Hard drives also become much less efficient when the disk is nearly full or if a Windows disk has not been recently defragmented.  Network disks and SANs may be many times slower than directly attached storage.  RAIDs must be carefully configured to provide maximum performance, see Tech Note 0018.  See the ExpeDat documentation for more information about storage performance.
Memory Cache
Most operating systems will use a portion of main memory (RAM) to store data that has been recently read from or written to disk.  This can reduce the affect of Disk Speed and substantially increase performance.  This also means that the first time a file is transferred may be much slower than subsequent attempts.  Increasing the amount of available memory may increase performance.
Enabling ExpeDat's inline compression feature will cause both the client and server to consume many times more CPU resources.  If the CPU load is the limiting factor, then enabling compression may cause the transfer to go slower, even if the data being transferred is compressible.  Inline compression also limits MTP's ability to cope with packet-loss on high-latency networks, which may reduce performance on those networks even when CPU load is not a problem.  See Tech Note 0014 for more about the effects of compression.
Encrypting and decrypting data requires extra CPU processing on both the client and server.  If the CPU load is the limiting factor, turning off encryption may improve performance.  Note that ExpeDat always encrypts usernames and passwords, even if content encryption is turned off.
If server-side packaging utilities are used, ExpeDat will be unable to move data any faster than those utilities deliver it.  Likewise, if the client-side output is piped to another application, ExpeDat will be unable to move data any faster than the other application can process it.  Packaging will greatly increase CPU and memory requirements, which may also affect performance.

Factors Affecting MTP Tunnel Performance

The MTP Tunnel application uses MTP/IP to move data between a TCP/IP based server and a TCP/IP based client.  To do this, MTP Tunnel must maintain TCP/IP connections to the server and client at both ends, and copy the data between TCP/IP and MTP/IP.  This introduces several additional factors which may affect performance.

Maximum Server Throughput
MTP Tunnel cannot move data faster than the TCP server sends it.  If the server is limited in the rate at which it sends data, this will limit the rate at which MTP Tunnel can move the data.  Be sure that the server is not limiting the rate at which it delivers data.
Client Throughput
MTP Tunnel cannot move data faster than the client consumes it.  If the client is unable to process incoming data quickly, MTP Tunnel may be forced to pause and wait for the client to be ready.
Copy Overhead
MTP Tunnel must copy data from TCP to MTP at the server side, and then copy the data back from MTP to TCP at the client side.  This causes a delay which may affect performance.  If TCP and MTP performance are close due to MTP/IP specific factors, this extra copying time may make MTP Tunnel slower than ExpeDat or plain TCP.
MTP Tunnel must copy data from TCP to MTP at the server side, and then copy the data back from MTP to TCP at the client side.  If the network is very fast, or if the Web Browser is very slow, larger buffers may be needed.  If you are using MTP Tunnel on a network faster than 100 megabits per second, you may need to increase its buffer size.  See the MTP Tunnel documentation for details.
Dropped Connections
If either the server or client experiences a problem, it may drop its connection to MTP Tunnel.  If this happens, MTP Tunnel will be forced to stop the transaction.  If you are experiencing dropped connections, investigate the client and server for error messages.

Performance Checklist

If MTP/IP appears to be performing poorly in a particular environment, perform the following checks to determine the cause.  You should read through the entire list before beginning, because some checks may be easier or faster than others to perform.  Most of these checks can be performed using the software included in your MTP/IP product..

Enable diagnostic output in the software which is receiving data.  This is typically done by setting "Debug 1" or "-d 1".  Perform a data transfer lasting at least 30 seconds and watch for errors and warnings.  At the end of the transfer, look for a block of data labeled "Statistics".  Submit this block to Tech Support for analysis.
Check your billing or service records to verify the intended capacity of the data lines at both the client and the server.  Remember that some links, such as DSL and cable, have different upload and download speeds.  Write down the speeds, link type, and IP addresses.  Compare this to the test results.  Beware of units errors: don't try to compare bits to bytes.  Use independent timing, such as a stopwatch, to measure performance.  Note what percentage of the link capacity is being used by each protocol.
Use the tracert or traceroute commands to measure latency and the number of hops between the client and servers.  If you cannot get through in BOTH directions between the client and server, but other traffic does get through, then you KNOW there is a firewall blocking traffic in between.  You can use the mtping utility to detect firewalls which are specifically blocking MTP traffic.
Firewalls, Emulators, NATs, Routers, etc.
Make SURE that the tests are not being affected by firewalls, NAT, or VPN devices.  If possible, disable any software firewalls, NAT programs, VPNs, or other packet filters which may be running on the client or server.  CHECK the configuration of all routers, hubs, DHCP devices, NAT devices, VPNs, and modems to make sure their firewall features are disabled or configured to EXPLICITLY allow BOTH incoming and outgoing traffic and the ports used by your MTP/IP application.  The test utility uses port 8082 by default.
VPNs and Tunneling
If a VPN is being used, check its documentation and configuration to verify that it DOES NOT tunnel UDP/IP over TCP/IP.  Devices which tunnel network traffic over TCP/IP, including "SSL VPNs" severely impair performance and are not compatible with MTP/IP.  Consider using an IPsec VPN instead.
Make SURE that none of the network devices is performing compression.  To test for network compression, build two test files each large enough to take at least 30 seconds to transfer.  One should consist entirely of zeros, the other should consist entirely of random data.  If the file consisting of zeros transfers much faster than the one consisting of random data, then something is compressing data in the network.  If you don't have an easy way to create zero and random data, using a plain text file and a compressed binary file will also work.
Network Service Provider
Check with both your client AND server network providers to see if they are (a) blocking any ports, (b) using any kind of network compression technology, (c) using any kind of packet shaping, rate throttling, or bandwidth management technology.  If possible, have them disable these technologies, at least for the purposes of testing.
Check that you are using modern networking hardware from reputable vendors.  Older or low-quality equipment may not conform to Internet Protocol standards.  If possible, try testing with different equipment such as CPUs, routers, and modems to see if a particular piece of hardware is causing the problem.

Contact DEI

Feel free to request technical support if you are unable to resolve any performance issues.  Be sure to let DEI's engineers know what steps you have already taken to isolate the problem.

Tech Note History

Feb282013Updated Test Instructions
Dec022011MTP Tunnel
Oct262010Tech Note 0024
Worksheet Link
General Cleanup
Feb172009ExpeDat Compression
Feb122007Adapted from PDF