Object Handlers

Clients can request that servedat launch external processes to perform actions or access external data storage.  An Object Handler can be used to perform tasks such as processing data, sending notifications, or moving files.  It can also be used to stream data to or from other applications.  For example, an Object Handler could be used to transfer data directly into or out of a database.  Object Handlers allow any back-end storage or processing to be tied to an ExpeDat server and controlled by ExpeDat clients.

This section applies to ExpeDat.  The SyncDat client does not currently support Object Handlers.

ExpeDat's inline compression and Streaming Folders features cannot be used with an Object Handler, but the handler itself may perform compression, packaging, or similar functions.  For example, a movedat client might pipe data through tar and call on the server to do the same with an Object Handler.

Setup

To make a server-side program available to ExpeDat clients, it must be declared using the –o option or ObjectHandler configuration variable.  Each object handler is assigned an "action code", which clients will use to designate its use.  Clients can only run programs which have a declared action code.  For example, servedat.cf might declare:

ObjectHandler Notice,/usr/local/libexec/sendnotice.sh

The movedat client could be used to run this handler using the following syntax:

echo "This is a message for Joe" | movedat user@example.com=Notice:joe

The command above would cause servedat execute /usr/local/libexec/sendnotice.sh.  That script would receive "This is a message for Joe" via stdin and the string "joe" via an environment variable.  The script could then send an email to "joe" containing the message.  A example of sendnotice.sh is shown below.

Access Restrictions

All users may access an object handler by specifying the appropriate action code.  Some users may be restricted to only accessing object handlers, not the filesystem, via the global ObjectOnly or AuthFile ObjectOnly restrictions.

The state of some server access privileges and user options are passed to the object handler through its environment.  It is the responsibility of the object handler to decide what restrictions to apply and how.  For example, the CloudDat S3 object handler uses RestrictHome to indicate that the user's home path should be prepended to the object key name.

Structure

Object handlers must conform to the following requirements:

When servedat receives a client request declaring a known Action Code, it will attempt to launch the corresponding program.  If the transaction was authenticated, the executable is launched with the user identity associated with the given username.  The transaction will fail if that user does not have permission to run the executable.

Once running, the handler should examine SV_TYPE to determine whether this is a GET or SEND transaction, and SV_ACTION to determine what task it should perform.  It can then perform the appropriate tasks and either read from stdin (for SEND) or write to stdout (for GET).  Two-way communication with the client (both stdin and stdout) is not available.

Following is a template for the bash shell showing a handler structure supporting multiple ExpeDat functions:

#!/bin/bash # Identify the handler and sub-action codes IFS=',' read -r -a ACTS <<< "$SV_ACTION" # ${ACTS[0]} is the identifier used to trigger this script # ${ACTS[1]}, if present, is an action code such as: *dl, *mk, or *mv if [ "$SV_TYPE" == "GET" ]; then if [ "${ACTS[1]}" == "" ]; then # Plain Get - send the path contents to stdout /bin/cat "${SV_PATHRAW}" elif [ "${ACTS[1]}" == "*dl" ]; then # Delete - delete the indicated path /bin/rm "${SV_PATHRAW}" elif [ "${ACTS[1]}" == "*mv" ]; then # Move - rename the path /bin/mv "${SV_PATHRAW}" "${SV_ARG}" elif [ "${ACTS[1]}" == "*mk" ]; then # MkDir - Create a folder /bin/mkdir "${SV_PATHRAW}" else # MTP_APP_FEATURE echo "Unsupported action: ${ACTS[1]}" exit 28 fi else # Send - Process data arriving on stdin /bin/cat > "${SV_PATHRAW}" fi; if [ $? -ne 0 ]; then exit 29 # MTP_APP_SYSTEM fi exit 0

For a plain data transfer, the SV_ACTION environment variable contains just the action code associated with the Object Handler.  When a standard ExpeDat client is called upon to perform a task such as deletion or folder creation with an Object Handler, it will invoke a GET transaction and append a secondary action code.  For example, the following two movedat commands are equivalent:

movedat -x user@example.com:deleteme=MyHandler movedat 'user@example.com=MyHandler,*dl:deleteme'

The handler example above shows just one possible structure for handling file transfers.  Any program meeting the requirements above, not just shell scripts, can be used.  Any action codes codes might be supported.  Any type of data or processing can be handled.

The example above performs only limited error checking.  A more complete implementation might enforce access restrictions or attempt to translate error conditions into more precise MTP_APP constants.  All input strings should be handled carefully to ensure that unexpected characters cannot produce security problems.

Environment

The handler program is launched directly, without shell interpretation.  It should expect no command line options or shell environment variables.  The only communication with servedat is from the servedat environment variables described below, the executable's exit code, and either stdin or stdout.  The program will be launched with the system privileges associated with the given username, if any.

The Object Handler is responsible for interpreting and enforcing Access Privileges based on the information provided in the environment variables below.

Some command line utilities depend on the existence of HOME or PATH environment variables.  The Object Handler must create these variables if they are needed.  In some cases, such as an AuthFile user, suitable values may not exist.

Windows based object handlers must pay special attention to the formatting of text and paths.  All environment variables are Unicode and all paths are forward-slash delimited.  When using a command shell as an object handler, set its code page to UTF-8 with the command "chcp 65001".

If PackTimeout is set, then the executable will be terminated by servedat if it runs longer than the specified time.  Otherwise, the transaction with the client will remain open until the executable exits, the client aborts, or a network error occurs.  In the event of an abort or error, servedat will attempt to terminate the handler and its descendants.

The following variables are provided in the handler program's environment.  Those marked "optional" may not be present if the corresponding value is not set.

Name Value
SV_ACTION <Action>[,Secondary_Action]
The first string is the Action Code used to activate this executable.  This may be followed by a comma and a secondary Action Code.  Clients may use the secondary Action Code to request file maintenance actions such as deletion or folder creation.  You may use movedat or the ExpeDat Client SDK to specify any string for SV_ACTION.  See below for handling '*li' or '*lr' listing requests from ExpeDat Desktop.
SV_ARG <String>
Optional An additional string provided by the client.  movedat and ExpeDat Desktop will use this to specify the target of a move action.  The ExpeDat Client SDK may specify any string.
SV_AUTHTYPE NONE, PRIVATE, or SYSTEM
The authentication mechanism used to validate this transaction.  NONE indicates that the server is operating in Quick Start mode.  PRIVATE means the username matched an AuthFile entry.  SYSTEM means the username matched a System User.
SV_BULK 0 or 1
When present and non-zero, this indicates that the client has declared this to be a "bulk" transaction for the purposes of logging and session identification.
SV_CLIENTIP <IP Address>
The client IP address, as seen by the server.  If the client is behind an NAT gateway, this will be a public address of that gateway.
SV_CLIENTPORT 1 to 65535
The UDP port of the client, as seen by the server.  If the client is behind an NAT gateway, this will be a public port of that gateway.
SV_DOCID <Hexadecimal>
An 8 character hexadecimal number which uniquely identifies this transaction.  It corresponds to the "DocID" field in the log file.
SV_ENCRYPT AES
Optional If present, this variable indicates that content encryption is being used with the named type.  Currently, only AES is supported.
SV_FEATURES <String>
Optional If present, this variable contains a comma separated list of transaction feature codes requested by the client.  Unknown or unsupported features should be ignored.
SV_GETONLY 0 or 1
When present and non-zero, this indicates that this transaction is subject to the GetOnly restriction.  The Object Handler is responsible for interpreting and enforcing this restriction.
SV_HEADER <Listing Header>
A structured listing header line, describing the current state of the server.
SV_HOME <Pathname>
Optional If present, this string contains the Home Directory associated with this transaction.  When SV_AUTHTYPE is SYSTEM, this is likely (but not guaranteed) to be equivalent to a shell $HOME path.
SV_LICENSE <String>
A description of this server's software license.  See Tech Note 0001 for more information about interpreting identification strings.
SV_LOGPREFIX DocID SEND|GET Client User Encrypt "Path" "Action"
A pre-formatted log line using the same format as that of I and F log file records.
SV_MAXLOADSTART <Integer>
The maximum server load since startup.
SV_MAXLOADCONFIG <Integer>
The maximum server load since the last configuration reload.
SV_NOOVERWRITE 0 or 1
A value of 1 indicates that this transaction is subject to the NoOverwrite restriction.  The Object Handler is responsible for interpreting and enforcing this restriction.
SV_PATHFULL <Target Pathname>
Not Available for Object Handlers The full pathname of the target file, after accounting for home directories, access privileges.
SV_PATHRAW <Target String>
The raw target string as given by the client.  Although considered by clients to be the "remote path", this can be any UTF-8 string and is not interpreted or modified by the server.  It is not guaranteed to be valid unicode.
SV_READONLY 0 or 1
A value of 1 indicates that this transaction is subject to the ReadOnly restriction.  The Object Handler is responsible for interpreting and enforcing this restriction.
SV_RESTRICTHOME 0 or 1
A value of 1 indicates that this transaction is subject to the RestrictHome restriction.  The Object Handler is responsible for interpreting and enforcing this restriction.
SV_SERVEDAT servedat - 1.16.6 August 2015 - DEI, DOC-2.1.2 MTP-osx-4.1.3 1604
The server's identification string.  See Tech Note 0001 for more information about interpreting identification strings.
SV_SERVERLOAD 0 to Capacity
The number of simultaneous transactions currently being served, including this one.
SV_SERVERCAP <Integer>
The maximum number of simultaneous transactions supportable by this server.
SV_SIZE <Bytes>
Optional For SEND transactions where the size of the source is known, the number of bytes expected.
SV_TIMEOUT <Seconds>
Optional If PackTimeout is set, then this variable will be present and contain the maximum number seconds the script will be allowed to run.
SV_TYPE GET or SEND
The type of transaction.  GET indicates that data sent to stdout will be delivered to the client.  SEND indicates that data from the client must be read from stdin.
SV_USER <Username>
Optional The username given by the client and authenticated by the server.  A value which is blank or "ANONYMOUS" indicates that authentication is enabled and no username was given.  If this variable is not set, then authentication is not enabled and this was an implicit anonymous request.
SV_WRITEONLY 0 or 1
A value of 1 indicates that this transaction is subject to the WriteOnly restriction.  The Object Handler is responsible for interpreting and enforcing this restriction.
SV_WRITELISTONLY 0 or 1
A value of 1 indicates that this transaction is subject to the WriteListOnly restriction.  The Object Handler is responsible for interpreting and enforcing this restriction.

Additional environment variables are likely to be introduced in future releases.  Please let us know if you have suggestions.

Support for Listings

To fully support an ExpeDat Desktop client, an Object Handler must respond to requests for a listing of available objects.  For a handler providing objects to download (GET), this must be a list of available objects.  For a handler which allows uploads (SEND), this might be a list of objects already uploaded.  The listing will be displayed in the Remote browser pane of the ExpeDat Desktop window.

ExpeDat Desktop requests a listing by sending a GET transaction to the handler with the secondary action "*li".  The handler must respond with the contents of SV_HEADER, followed by the list of objects (if any) in the ExpeDat Structured List format.  All lines must end with a single Line Feed character and all pathnames must be UTF-8 encoded.

The "Free Space" field of SV_HEADER may not be accurate for your handler.  Modify it to show a relevant value or erase it if a value is not known.

If the listing includes objects marked as directories, then ExpeDat Desktop may send a '*lr' request for a recursive listing of the directory in preparation for a download of the entire directory's contents.  If your handler is to support such a function, the listing should be returned in the same format as '*li', in depth-first order.

movedat will also request a "*li" secondary action when run with an object handler, no destination, and a path that is either empty or ends in '/' (the usual syntax for listing a directory).  Supporting listings is not essential for movedat.

Performance

Data streamed through stdio will be buffered as per the StreamSize setting.  Because of this buffering, clients may not accurately report progress and cannot estimate a time of completion for GET transactions. 

Because of the overhead of launching the Object Handler program, the response time for even simple actions may be significantly longer than an equivalent action handled entirely within the server.  If the Object Handler stalls or blocks during its processing of data, overall server performance will not be impaired.  Only the throughput of the individual transaction will be slowed by bottlenecks in the Object Handler.  Data processing which is bursty or uneven may benefit from a larger StreamSize buffer.  For example, if it takes significant time for an Object Handler to prepare for incoming data, use a buffer large enough to absorb arriving data for that time.

Example

Below is an example of the "sendnotice.sh" email notification Object Handler referenced above.  Unlike the general purpose example, this script only accepts SEND transactions and treats the path as an email address rather than a filesystem path.

#!/bin/bash # servedat object handler for sending an email - 130715 # # The recipient address is the SV_PATHRAW. # The message body is read from stdin. # The path to "mail" varies by operating system. MAILER=/usr/bin/mail # Only support SEND transactions if [ "$SV_TYPE" == "GET" ]; then echo "This Object Handler only works with Send transactions" exit 28 # MTP_APP_FEATURE fi # Send the email $MAILER -s "ExpeDat Email Notice" ${SV_PATHRAW} STATUS=$? if [ $STATUS -ne 0 ]; then exit 29 # MTP_APP_SYSTEM fi exit 0

This handler could be declared in the servedat configuration as follows:

ObjectHandler Notice,/usr/local/libexec/sendnotice.sh

It would then be run by movedat like this:

echo "This is a message for Joe" | movedat user@example.com=Notice:joe

A more sophisticated implementation might try to translate the exit codes of the "mail" command, perform user or input validation, or lookup recipient aliases.