Packaging

In addition to directly accessing files, servedat can be configured to plug-in third-party archiving utilities.  This allows servedat to produce output in other formats, such as tar, zip, cpio, or even retrieve data from other applications such as mysql.

This section applies primarily to ExpeDat.  The SyncDat client does not currently support Packaging.

To work with servedat, a utility must be capable of running from the command-line and sending its output to stdout.  Tar, Gzip, and Info-Zip are examples of archiving utilities known to work with servedat.  Some Zip variations are known to not support stdout and so are not compatible with servedat.

Packaging allows servedat to work with software produced by third-parties.  The examples below are only guidelines.  Carefully check the documentation for any third-party software you intend to use with servedat.

Syntax

Adding support for a packaging utility is done with the Package configuration variable or the "–P" command line option:

-P <suffix>,'"<command>" <options> "%s"'

Note the careful arrangement of single and double quotes on the command line to ensure correct handling by the command-shell.  <suffix> is the file name extension you wish to associate with the given package format.  <command> is the path name of the command line utility which will be used to generate the package.  <options> provides any command switches needed by the utility to create an archive and output it to stdout.  The string "%s" will be replaced with the path name of the directory or file to be packaged at the time of download.

Running Other Applications

This plug-in facility can be used to execute any application you choose, not just archiving utilities.  As long as the application takes a pathname as an argument, you can use Packaging to run it and retrieve its output.  This can be useful for post-processing uploaded files.  See DEI Technical Note 0015 for more integration ideas.

Examples

In Windows, <command> should be the fully qualified path name to the command executable.  It is not subject to command-shell parsing, so shell special characters may not be interpreted.  Following is an example using Info-Zip under Windows:

-P zip,"\"C:\Program Files\Zip\zip.exe\" -q -r - \"%s\""

All other platforms will execute the entire command string using /bin/sh.  This means that shell special characters will be interpreted and multiple command utilities can be pipelined together.  Below are some unix examples:

-P tar,'/usr/bin/tar -cf - "%s"'
-P tgz,'/usr/bin/tar -cf - "%s" | /usr/bin/gzip -c'
-P zip,'/usr/bin/zip -r -q - "%s"'
-P zip,'/usr/bin/ditto -c -k -rsrc "%s" -'

For security reasons, characters which may have special meaning to a command shell are not permitted in the names of files to be packaged.  It is also recommended that you use a fully qualified path name for each utility.

On some platforms, use of GNU Tar's built-in Gzip ability, "tar -zcf", is not compatible with servedat.  Instead, pipe the tar output through Gzip as shown above.

You may specify as many packaging commands as your system can support by adding additional "–P" options.  For example, to support both tgz and zip in unix:

servedat -P tgz,'tar -cf - "%s" | gzip -c' -P zip,'zip -r -q - "%s"' .

Likewise, you may specify multiple Package variables in the configuration file.

Usage

Clients requesting a packaged download will supply a list of suffixes for the server to choose from.  servedat will choose the first supported format from the client's list.  When you are choosing which packaging systems servedat should use, consider the type of data you are transporting.  Files which are already compressed or do not compress well may transfer much more slowly if you try to compress them again.  Such files should use a non-compressing format like Tar alone or Zip with compression turned off (zip -0).  Also note that using packaging plug-ins will consume much more of the server's memory and CPU time.  This may impact network performance on slow or heavily used machines.

servedat normally lets packaging programs run until they complete.  However, some third-party packaging processes may hang or run for excessive periods of time.  You can limit the run time of the third-party packaging programs by using the "-E" option or setting the PackTimeout configuration variable.  servedat will then terminate any process which runs longer than the given amount of time.

Downloads

If your system does not have a packaging utility installed, you may wish to try one of the following freeware utilities available from third parties:

If you are having trouble locating a compatible Zip binary for Windows, contact DEI Technical Support to receive an Info-Zip executable.