Data Expedition, Inc. ®

Move Data Faster

Support

Support
Tech Notes
General Information
Software Versions
Platform Support
Purchase Order Info.
Updating Software
ExpeDat Integration
Integration Strategies
Deprecated Platforms

Integration Strategies

Page Index:
Embedding
ExpeDat Client SDK
MTP Tunnel
Document Exchange SDK
Sequencing SDK
Core MTP/IP SDK
Choosing
Tech Note History
Nov042011MTP Tunnel
Apr202009First Post

MTP/IP Integration Strategies

The purpose of this article is to assist developers in choosing which interfaces are most suitable for integrating MTP/IP network acceleration technology into any given application.  The MTP/IP technology is designed for integration into the widest possible variety of operating system and software development platforms.  This flexibility is supported by several different interfaces which can be used alone or in combination.

Following is an examination of the various MTP/IP interfaces that are currently available.

Embedded Executables

The simplest way to integrate MTP/IP technology into an existing application is to embed an off-the-shelf DEI product alongside the target application.  All DEI products are available in the form of small, single executables which can easily be called from scripts, batch files, or other executables.

ExpeDat, in particular, has many features specifically designed to enable tight integration through its command-line interfaces.  These features include detailed exit codes, machine parseable logs and progress updates, standard I/O streaming, and more.  See Tech Note 0015 for details about Embedding ExpeDat.

Primary Target
Existing file transfer applications or scripts, particularly those which already call out to FTP or SCP executables for accessing remote server software.
Advantages
Rapid plug-in deployment, sometimes in as little as 5 minutes.  Little or no modification to legacy source-code.
Disadvantages
Only suitable for transferring files or standard I/O data streams.  Requires installation of ExpeDat client and/or server executables.

ExpeDat Client SDK

The ExpeDat Client Software Development Kit (EXP) is a C language library which can be used to build custom ExpeDat client software.  DEI's own ExpeDat and SyncDat clients are built using EXP, giving your application access to the same functionality as our own.

EXP provides simple, high-level interfaces for communicating with an ExpeDat server.  This includes sending and receiving files, authentication, encryption, directory listings, file maintenance, and even streaming file content to or from memory.  Once the parent application starts a file transfer, it does not need to perform any further actions until it is notified that the transaction has completed.

Primary Target
New or existing applications whose primary function is file transfer.  This includes not only traditional FTP style clients, but also managed file transfer systems (MFTs), content delivery networks (CDNs), and any application which may move files along a client-server model.
Advantages
Source code level integration allows the highest level of performance and the most user interface control and flexibility.  Data can be delivered to or from any source, including disk, memory, or streams.
Disadvantages
Requires installation of an ExpeDat server component at the server side.  Focused on client-server file transfer and so not suitable for general data exchange between peers.

MTP Tunnel

The MTP Tunnel port forwarding utility captures connections from TCP clients to TCP servers, transporting the data across the network using MTP/IP as an intermediary.  This allows you to accelerate off-the-shelf and legacy TCP applications without changing the existing software or workflows.  Any client-server TCP session involving large amounts of continuous data flow can be captured by directing it to an outgoing tunnel port, similar to Secure Shell port-forwarding.  Client applications can also be forwarded via stdio for embedded solutions.

Primary Target
Legacy and off-the-shelf TCP applications performing high-volume data transport.
Advantages
Works with existing TCP clients and servers, regardless of purpose or data type.
Disadvantages
Only suitable for high-volume, continuous data flows.  Applications which produce data slowly or are "chatty" may not be accelerated.  Requires significant RAM when used on high-speed data paths.

Document Exchange SDK

The Document Exchange Software Development Kit (DOC) is a C language library which provides session management and data delivery for general purpose data transfer between networked applications.  DEI's own end-user products are built on top of DOC, providing a consistent platform for inter-application communication.

The DOC SDK is focused on the idea of sending or retrieving blocks of data which may be located on disk, in memory, or in sequential streams (see Sequencing below).  It provides a high-level model for both client and server networking, allowing the parent application to focus on data management.  Once an application initiates or accepts a document transfer, it does not need to perform any further actions until it is notified that the transfer has completed.

Primary Target
General data transfer between applications, including client-server, peer-to-peer, and one-to-many.
Advantages
Complete integration and control over all components, regardless of communication model.  "Fire-and-forget" data transfer minimizes the need for network level design awareness.  Data can be delivered to or from any source, including disk, memory, or streams.
Disadvantages
Developer is responsible for providing all data management functionality, such as authentication, data storage, user interfacing, encryption, and so on.

Sequencing SDK

The MTP Sequencing Software Development Kit (SEQ) is a C language library which allows data being delivered via MTP/IP to be buffered and accessed using a read/write data model.  This allows program generated data to be sent, or incoming data to be processed sequentially and entirely within memory.  DEI's own end-user products use SEQ for tasks such as compression and decompression, listing directory trees, and streaming standard I/O.

The SEQ SDK can be used as a data target for any of the other DEI Software Development Kits, including the ExpeDat Client SDK, Document Exchange, and Core MTP/IP.  It can also be used for general purpose intra-process buffering, independent of the network.  In addition to basic read/write functionality, it also provides asynchronous notification, random buffer access, flow-control, and the ability to examine buffer capacities.  If you are used to programming against TCP/IP sockets, the SEQ interface will be at once familiar and liberating.

Primary Targets
Supplement for streaming data in applications using other MTP SDKs.  Legacy interface for existing applications written against TCP/IP.
Advantages
Traditional read/write access.  Many sophisticated features that are lacking in typical sockets style interfaces: asynchronous notification, full buffer access, peek/poke, and more.
Disadvantages
Each SEQ construct requires allocation of a memory buffer that is typically several megabytes in size.  Networks which are very fast and/or very high-latency may suffer performance limitations if the buffer is inadequate.

Core MTP/IP

The Core MTP/IP Software Development Kit (Core MTP) is a C library providing access to the most basic building blocks of DEI technology.  MTP/IP transactions are highly optimized to deliver maximum network performance with minimum overhead.  For example, an entire network transaction can be reliably completed in a single round-trip time with no setup or tear-down overhead.

Implementing on top of the Core SDK delivers maximum efficiency in both network and CPU utilization, and maximum flexibility to achieve that efficiency for almost any network communication task.  However, with that flexibility comes a great deal of design responsibility.  Developers working directly with Core MTP/IP need to have a strong understanding of how packet-switched IP networks operate and experience designing network applications.  MTP/IP will handle the details of flow-control and error-recovery, but it is up to the developer to decide issues such as session management, authentication, data processing, and so on.

Primary Target
Applications requiring rapid transaction processing, precise network control, and the tightest possible source code integration.
Advantages
Complete freedom to create a highly optimized network application.
Disadvantages
Requires an advanced understanding of network transaction processing and software design.

Choosing

Most MTP/IP integrators find it simplest to begin with Embedded ExpeDat.  This provides the fastest integration path and, for most file transfer applications, it provides more than adequate functionality.  For projects which require maximum client-side integration, the ExpeDat Client SDK provides source-code level control without requiring the programmer to delve into network level details.

Developers seeking to create or accelerate general purpose network communication paths will find that the Document Exchange and Sequencing SDKs provide that functionality with minimal network level consideration.  The Core MTP/IP SDK is for advanced network programmers who wish to create their own highly-optimized communication software.

Integrators seeking to accelerate legacy applications, where source code is not available and ExpeDat's file-transfer model does not fit, may consider MTP Tunnel as a way to capture and accelerate existing TCP/IP sessions.

For more information about MTP/IP integration options, including a consultation with one of our engineers and a free trial of any SDKs, contact us.