Embedding ExpeDat

The ExpeDat file transfer application can be used to do much more than transfer files under end-user control.  The light-weight command-line client and server applications are designed to easily interact with other software.  This not only allows ExpeDat operations to be automated by other processes, but enables complex interactions between ExpeDat and other programs.  All of this can be achieved without source-code level integration, allowing for fast, modular deployment.

Below are the most common examples of how ExpeDat has been embedded and integrated with other software.  Please note that many of these examples are fragmentary, so a strong familiarity with your operating system and general programming techniques is assumed.  In particular, be aware of the varying need to escape or quote special characters which may have significance for the shell.

Web Integration

For Windows and Mac OS X systems, ExpeDat transfers can be started from web pages, emails, PDFs, or anywhere else you can click a link.  Transfers are initiated using the "expedat://" URL scheme.  See the Web Integration chapter of the ExpeDat documentation for details.

A JavaScript API is also available for programmatically creating and executing "expedat://" URLs.  See the JavaScript API chapter of the ExpeDat documentation for details.

Shell or Batch Scripts

The simplest way to automate ExpeDat operations is to wrap them in a shell script (or a batch script in Windows).  For example:

#!/bin/sh # Take argument $1 as the root of two files to download. movedat "server:$1.part1" /local/stagingdir/ movedat "server:$1.part2" /local/stagingdir/

Many of the techniques described below show more complex ways that ExpeDat can interact with scripts.

cron or AT Jobs

You can use the unix cron service or the Windows AT service to automate execution of ExpeDat commands at regular intervals.  The following cron job would download a particular file every weekday at 5:00am:

00 05 * * 1-5 movedat server:/path/file.jpg /home/me/pictures/

The following AT command could achieve the same thing in Windows:

AT /every:m,t,w,th,f C:\ExpeDat\movedat.exe server:/path/file.jpg "C:\Pictures\"

exec, popen, or CreateProcess

In addition to shell scripts, native programs can also call on movedat.  In unix, this is often done with the exec() or popen() system calls.  In Windows, this may be done with the CreateProcess() call.  For example:

pipefd = popen("movedat server:/pathname/file /local/dir/","r+");

Stored Authentication

When calling movedat from a script or program, it will be unable to prompt a user to input a password.  Therefore there are two methods that you can use to supply a password programatically.  The simplest method is to include the password on the command line:

movedat user:password@server:/pathname/file /local/dir/

Alternatively, movedat can store passwords in an encrypted cache for later use.  For example:

movedat -A user@server:

or

movedat -A user:password@server:

The first example will prompt you to enter a password, while the second will take it from the command line.  Both examples will encrypt the password and store it.  If a command later refers to "user@server:", the encrypted password will be used automatically.  See the movedat Authentication chapter of the ExpeDat documentation for more details.

Completion Status

The movedat client returns a numerical exit code which follows the unix convention of 0 for success, and non-zero error codes for failures.  Many error codes are supported, allowing a script or parent software to differentiate between problems.  For example, code 70 indicates a network problem and so it may be desirable to try again later.  Code 10 indicates a bad username or password, and so it may be desirable to query a human user for the correct credentials.  See the movedat Exit Codes documentation for details.

Progress Tracking

When movedat is run underneath another program, the "-s" option will cause it to display machine parseable logs for each operation.  This allows the parent software to monitor progress and errors, and provide user feedback.  See the movedat Logging documentation for details.

Data Streaming

In addition to transferring named files, movedat can be used to "pipe" data to or from another program.  For example, the following shell command would create a tar archive on the fly and upload it to a server:

tar -zcf - /path/dir/ | movedat - user@server:dir.tgz

The next example shows how incoming data can be unpacked as it is downloaded:

movedat user@server:dir.tgz | tar -zxf -

When combined with exec, popen, or CreateProcess, this technique can be used to feed data directly into or out of a parent application.  See the movedat Piping Files documentation for details.

Package Scripts

The servedat server can also interact with other software, through the package plug-in mechanism.  This allows data to be retrieved from remote programs executed on the server.  The most common example is to use packaging software such as tar.  The server could be started as

servedat -P tgz,'/usr/bin/tar -cf - "%s" | /usr/bin/gzip -c'

and then the client run as

movedat user@server:/dir/to/pack=tgz /home/me/localdir/

This can be combined with data streaming to pack and unpack in a single operation:

movedat user@server:/dir/to/pack=tgz | tar -zxf -

See the servedat Packaging chapter of the ExpeDat documentation for details.

Action Scripts

Administrators can configure scripts or applications for post-processing, notifications, file injest, or any other purpose.  Action Scripts are invoked by the movedat client, using syntax similar to packaging.  Any command-line software can be used, and a variety of environment variables are available to provide inputs.

Suppose you wish to upload a file to the server, then have it processed by another application.  For example, you might have a program called "ingest" to import the file into a document management system.  This could be configured as a packaging plug-in:

servedat -P ing,'ingest "%s"'

The client could first upload the file, then trigger the ingestion:

#!/bin/sh movedat newfile user@server:/tmp/newfile if [ $? ] ; then movedat user@server:/tmp/newfile=ing > /dev/null ; else echo "Error $? occurred" fi

The second movedat command runs the ingest program on the server and returns its output.  More sophisticated error checking could be done by testing the exit code and piped output of that execution.

See the servedat Action Scripts chapter of the ExpeDat documentation for details.

Directory Listing

When movedat requests a directory listing, the server sends the results as a series of structured records which movedat then interprets and displays.  You can directly access these records, allowing another application to interpret and display them.

The following command signals movedat to list the files in the named directory, and output the raw records:

movedat user@server:pathname=\*li

You can then pipe this data to another application for interpretation, display, or searching.  See the Structured Lists section of the ExpeDat manual for details.

More Techniques

These are just simple examples of how ExpeDat can be integrated with other software.  Please refer to the documentation referenced in each section for more details.