Tech Note 0015
Methods for integrating ExpeDat with other applications, scripts, and work-flows
The ExpeDat file transfer software can be used to do much more than transfer files under end-user control. The light-weight command-line client and server applications are designed to easily interact with other software. This not only allows ExpeDat operations to be automated by other processes, but enables complex interactions between ExpeDat and other systems. All of this can be achieved without source-code level integration, allowing for fast, modular deployment.
Below are the most common examples of how ExpeDat has been embedded and integrated with other software. Where code examples are shown, note that syntax may very depending on your platform. In particular, be aware of the varying need to escape or quote special characters in different shell environments.
A custom URL scheme and client-embedded API server allow web pages and other documents to start and monitor ExpeDat transactions. When ExpeDat Desktop is installed, triggering expedat:// links will automatically launch the client and perform actions such as uploading or downloading files. While ExpeDat Desktop is running, a built-in HTTP API server allows webpages to communicate directly with the client to start and monitor transfers within the browser.
See the Web Integration chapter of the ExpeDat documentation for details on this and other web integration functionality.
servedat server can send or retrieve data using external programs without touching disk. This allows it to act as a gateway to any back-end data system such as cloud storage or databases, and perform real-time processing such as transcoding. Tasks such as listings, deletion, rename, and folder creation can be performed by an Object Handler, allowing you to create fully functional virtual filesystems out of any back-end data system. Object Handlers can execute automation scripts and perform such tasks as email notification, file relocation, or any other function.
Access is defined by the server administrator. Clients choose amongst available handlers and provide inputs such as query strings, supplemental credentials, or object names. Exit codes are passed all the way back to the client to ensure reliable automation and storage.
See the servedat Object Handlers chapter of the ExpeDat documentation for details and examples.
Just as the server can stream files to and from object handlers, the
movedat client can be used to "pipe" data to or from client-side programs. For example, the following shell command would create a tar archive on the fly and upload it to a server:
tar -zcf - /path/dir/ | movedat - email@example.com:dir.tgz
The next example shows how incoming data can be unpacked as it is downloaded:
movedat firstname.lastname@example.org:dir.tgz | tar -zxf -
Shell or Batch Scripts
The simplest way to automate ExpeDat operations is to wrap them in a shell script (or a batch script in Windows). For example:
#!/bin/sh # Take argument $1 as the root of two files to download. movedat "server:$1.part1" /local/stagingdir/ movedat "server:$1.part2" /local/stagingdir/
Many of the techniques described below show more complex ways that ExpeDat can interact with scripts.
cron or AT Jobs
You can use the unix
cron service or the Windows
AT service to automate execution of ExpeDat commands at regular intervals. The following
cron job would download a particular file every weekday at 5:00am:
00 05 * * 1-5 movedat server:/path/file.jpg /home/me/pictures/
AT command could achieve the same thing in Windows:
AT 05:00 /every:m,t,w,th,f C:\ExpeDat\movedat.exe server:/path/file.jpg "C:\Pictures\"
exec, popen, or CreateProcess
In addition to shell scripts, native programs can also call on
movedat. In unix, this is often done with the
popen() system calls. In Windows, this may be done with the
CreateProcess() call. For example:
pipefd = popen("movedat server:/pathname/file /local/dir/","r+");
movedat from a script or program, it will be unable to prompt a user to input a password. Therefore there are two methods that you can use to supply a password programatically. The simplest method is to include the password on the command line:
movedat user:email@example.com:/pathname/file /local/dir/
movedat can store passwords in an encrypted cache for later use. For example:
movedat -A firstname.lastname@example.org:
movedat -A user:email@example.com:
The first example will prompt you to enter a password, while the second will take it from the command line. Both examples will encrypt the password and store it. If a command later refers to "
firstname.lastname@example.org:", the encrypted password will be used automatically. See the movedat Authentication chapter of the ExpeDat documentation for more details.
movedat client returns a numerical exit code which follows the unix convention of
0 for success, and non-zero error codes for failures. Many error codes are supported, allowing a script or parent software to differentiate between problems. For example, code
70 indicates a network problem and so it may be desirable to try again later. Code
10 indicates a bad username or password, and so it may be desirable to query a human user for the correct credentials. See the movedat Exit Codes documentation for details.
movedat is run underneath another program, the "
-s" option will cause it to display machine parseable logs for each operation. This allows the parent software to monitor progress and errors, and provide user feedback. See the movedat Logging documentation for details.
movedat requests a directory listing, the server sends the results as a series of structured records which
movedat then interprets and displays. You can directly access these records, allowing another application to interpret and display them.
The following command signals
movedat to list the files in the named directory, and output the raw records:
movedat -D email@example.com=\*li:pathname
Custom reports can be injected into the servedat log file by using the *rp action code. For example, a report log can be generated using movedat like this:
movedat -D 'firstname.lastname@example.org=*rp:This is a report'
This will cause the server to record an R record including the report text. For example:
R 20220602 14:27:34.113 E897F076 GET 10.0.1.128:59545:E user AES "This is a report" "*rp"
The report text can be any UTF-8 string, but it is best to avoid quote marks, line feeds, and other formatting characters.
These are just some simple examples of how ExpeDat can be integrated with other software. Please refer to the documentation referenced in each section for more details.
Tech Note History
|Jun||02||2022||Client report logging|
|Jan||03||2021||Updated web integration|