Packaging
Only available with select server licenses.

servedat can be configured to retrieve data from external archiving utilities or other processes.  This allows servedat to produce output in formats such as tar, zip, and cpio, or to execute other third-party commands.

This section applies to ExpeDat.  The SyncDat client does not support Packaging.

To work with servedat, a utility must be capable of running from the command-line and sending its output to stdout.  Tar, Gzip, and 7-Zip are examples of archiving utilities known to work with servedat.  Some Zip variations are known to not support stdout and so are not compatible with servedat.

Packaging allows servedat to work with software produced by third-parties.  The examples below are only guidelines.  Carefully check the documentation for any third-party software you intend to use with servedat.

If you are simply looking for a way to transfer entire folders in a single transaction, consider the Streaming Folders functionality of MTPexpedat, movedat, and DropDat, or one of the Master modes of syncdat.

If you wish to create your own scripts or data processing utilities, see Object Handlers instead.  Packaging utilities differ from Object Handlers in the following ways:

Support for Packaging Utilities may soon be discontinued in favor of Object Handlers.  It is retained for now for compatibility with existing workflows.  If you have a workflow which depends on Packaging and need help migrating to Object Handlers, please contact us.

Syntax

Adding support for a packaging utility is done with the Package configuration variable or the "–P" command line option:

Package <suffix>,"<command>" <options> "%s"
  or
servedat -P <suffix>,'"<command>" <options> "%s"'

Note the careful arrangement of single and double quotes on the command line to ensure correct handling by the command-shell.  <suffix> is the file name extension you wish to associate with the given package format.  <command> is the path name of the command line utility which will be used to generate the package.  <options> provides any command switches needed by the utility to create an archive and output it to stdout.

The string "%s" will be replaced with the name of the directory or file targeted by the client.  The command will executed with a current working directory set to the folder containing the file.  Additional variations of the target path are available as environment variables.

You may specify multiple packaging commands by adding additional "–P" options on the command line or Package variables in the configuration file.

Environment

The command will be executed using the bourne-shell (sh) on unix systems and CreateProcess() on Windows systems.  The current working directory will be set to the parent folder containing the target file.  See Object Handlers for a description of available environment variables.

The server will verify that the target pathname exists and is accessible before attempting to start a Package command.

If PackTimeout is set, then the command will be terminated by servedat if it runs longer than the specified time.

Examples

In Windows, <command> should be the fully qualified path name to the command executable.  It is not subject to command-shell parsing, so shell special characters may not be interpreted.  Following is an example using Info-Zip under Windows:

-P zip,"\"C:\Program Files\Zip\zip.exe\" -q -r - \"%s\""

All other platforms will execute the entire command string using /bin/sh.  This means that shell special characters will be interpreted and multiple command utilities can be pipelined together.  Below are some unix examples:

-P tar,'/usr/bin/tar -cf - "%s"'
-P tgz,'/usr/bin/tar -cf - "%s" | /usr/bin/gzip -c'
-P zip,'/usr/bin/zip -r -q - "%s"'
-P zip,'/usr/bin/ditto -c -k -rsrc "%s" -'

It is recommended that you use a fully qualified path name for each utility.

On some platforms, use of GNU Tar's built-in Gzip ability, "tar -zcf", is not compatible with servedat.  Instead, pipe the tar output through Gzip as shown above.

You may specify as many packaging commands as your system can support by adding additional "–P" options.  For example, to support both tgz and zip in unix:

servedat -P tgz,'tar -cf - "%s" | gzip -c' -P zip,'zip -r -q - "%s"' .

Likewise, you may specify multiple Package variables in the configuration file.

Security

For security reasons, characters which may have special meaning to a command shell are not permitted in the target name.  If one of the following forbidden character appear in the path, either from the client or the home directory path, the action will fail with a pathname error.

" ' ; ` ( ) [ ] & ? * ! ~ | { }

In addition, the backslash character '\' is not permitted on unix systems.

Environment values for SV_ARG and SV_PATHRAW are provided directly by the client and are not validated by servedat in any way.

The script will execute with the same privileges as the user account used for authentication.  The server will advise of additional restrictions using SV_NOOVERWRITE, SV_READONLY, SV_GETONLY, SV_RESTRICTHOME, SV_WRITEONLY, and SV_WRITELISTONLY.  Since the server has no way of knowing what secondary effects a packaging command may have, these restrictions are not fully enforced by the server.

Because package commands and scripts may execute arbitrary commands, administrators should take care that they are not writeable by untrusted users.

Usage

Clients requesting a packaged download will supply a list of suffixes for the server to choose from.  servedat will choose the first supported format from the client's list.  When you are choosing which packaging systems servedat should use, consider the type of data you are transporting.  Files which are already compressed or do not compress well may transfer much more slowly if you try to compress them again.  Such files should use a non-compressing format like Tar alone or Zip with compression turned off (zip -0).  Also note that using packaging plug-ins will consume much more of the server's memory and CPU time.  This may impact network performance on slow or heavily used machines.

servedat normally lets packaging programs run until they complete.  However, some third-party packaging processes may hang or run for excessive periods of time.  You can limit the run time of the third-party packaging programs by using the "-E" option or setting the PackTimeout configuration variable.  servedat will then terminate any process which runs longer than the given amount of time.

Downloads

If your system does not have a packaging utility installed, you may wish to try one of the following freeware utilities available from third parties:

BsdTar for Windows http://gnuwin32.sourceforge.net/packages/libarchive.htm
7-Zip http://www.7-zip.org/
GNU Tar http://www.gnu.org/software/tar/
GNU Zip http://www.gzip.org/

Most unix systems have a variety of compatible packaging utilities pre-installed.  However, very few Windows packaging utilities support stdio data streaming.  Consider the Streaming Folders functionality of MTPexpedat, movedat, and DropDat as an alternative.