Advanced Config

Amazon Web Services

Each AWS instance of CloudDat includes a fully customizable ExpeDat server.  ExpeDat is DEI’s high performance file transfer software which uses our proprietary transport protocol to move data across the Internet, overcoming distance, latency, and congestion.

With a standard install, CloudDat for AWS provides access to a single S3 bucket for a single username.  Additional usernames, multiple buckets, access to EBS and instance storage, landing page customization, and full ExpeDat functionality can be enabled by modifying the CloudDat configurations.

Standard Setup

A standard install of the CloudDat on AWS uses the following ExpeDat configuration options.

  • The following line in /etc/servedat.cf ties the “S3” action code supplied by clients to the s3handler.sh shell script on the CloudDat EC2 instance.  See the Object Handlers documentation.

    ObjectHandler S3,/usr/local/expedat/s3handler.sh

  • AuthFile /etc/svpasswd
    ExpeDat users are defined in /etc/svpasswd, which resembles a classic unix password file.  The s3user created by the s3setup.sh script is given the ObjectOnly restriction, which means it cannot access the instance filesystem.  See the ExpeDat AuthFile documentation.  If you customize this file, be sure to answer ‘n’ if you re-run s3setup.sh when it asks about replacing existing credentials.
  • SysAuth 1
    This option in /etc/servedat.cf allows system accounts on the CloudDat instance, such as ec2-user, to access both S3 and the instance filesystem.  You must assign a password to a system account for it to be useable with ExpeDat.  For example, “sudo passwd ec2-user".  See the ExpeDat SysAuth documentation.
  • /var/www/html/index.php
    This is the CloudDat instance web page for downloading clients and monitoring server status.  The settings at the top of this file allow you to easily customize its look.  You can also rename it to allow other web content on the instance.

Following are some of the ways you can modify these settings to customize CloudDat.

Updating to a New Version

New versions of CloudDat for perpetually licensed software can be downloaded directly from your DEI Customer Site.  New versions of CloudDat from AWS Marketplace are obtained by launching a new CloudDat instance from AWS Marketplace and copying over configuration and other customer data from the old instance to the new one.

If you have performed any of the advanced customizations on this page, you will need to preserve the following files:

  • /etc/servedat.cf - Your old ExpeDat server configuration file may be preserved without modification, however you may wish to examine the new sample file for new features.
  • /etc/svpasswd - Your old ExpeDat password file may be preserved without modification, however you may wish to examine the new sample file for new features.
  • /usr/local/expedat/s3handler.sh - A new handler script must be installed as part of the update process.  If you made customizations to the handler script, be sure to save a copy and carefully replicate those customizations in the new script.
  • /var/www/html/index.php - Your old web landing page may be preserved without modifications, however you may wish to apply any customizations to the new version to gain new features.

General advice for updating DEI software can be found in Tech Note 0012.

Multiple Users

The standard setup creates a single S3 user.  To create additional credentials, edit /etc/svpasswd, using that first entry as a template.  For example, after adding a second user, svpasswd might look like:

s3user:frYypg…0HCxHA:99:99:/var/empty:ObjectOnly usertwo:SddKMR…jSHh7g:99:99:/var/empty:ObjectOnly

The second field of svpasswd can contain a plain text password, but it is safer to generate an SHA hash using the /usr/local/expedat/mkpasswd utility.  After modifying svpasswd, the server will automatically load the new credentials in about 2 minutes.

With multiple user credentials, you can apply different restrictions and configurations to different accounts, as discussed below.

Access Restrictions

The final field of an AuthFile (svpasswd) record contains a list of access restrictions.  Commonly used restrictions are ReadOnly (no uploads or changes allowed) and WriteListOnly (list, delete, rename, upload, but no download).  The ObjectOnly option prevents that user from accessing the instance filesystem.  The RestrictHome option can be used to restrict access within a bucket, as described next.

See the ExpeDat AuthFile documentation for more options.

Segmented Buckets

The default CloudDat user account, shown above, has access to the entire S3 bucket.  You can restrict gateway users to just a portion of the bucket by assigning a RestrictHome prefix.  For example,

s3user:frYypg…0HCxHA:99:99:webfolder:ObjectOnly,RestrictHome

This will insert the prefix “webfolder/” in front of all paths accessed by ExpeDat clients using the “s3user” name and will not allow them to access any other objects.  It is like restricting the user to a single folder.  You can create multiple users, each restricted to a different prefix.  Changes to svpasswd will be automatically loaded about two minutes after your last modification.

Multiple Buckets

Step 1 of the Install process asks you to choose an S3 bucket to access.  Setting a fixed name allows clients to access S3 storage without knowing the bucket name.  If you leave this blank, clients must specify a bucket name as the first part of the remote path.  This allows you to access multiple buckets through a single CloudDat instance.

If you’ve already setup the gateway, the easiest way to modify the bucket name is by editing /usr/local/expedat/s3handler.sh to change the “S3BUCKET=” setting at the top.

When S3BUCKET is blank, the bucket must be specified either by the client or as part of the svpasswd home prefix.  If you follow the Multiple Users and Segmented Buckets instructions above, you can have different users accessing different buckets all from the same CloudDat instance.

joanne:frYypg…0HCxHA:99:99:bucketone:ObjectOnly,RestrictHome robert:SddKMR…jSHh7g:99:99:buckettwo:ObjectOnly,RestrictHome samuel:Yd3751…PWFgIQ:99:99:bucketthree/sam:ObjectOnly,RestrictHome nancy:nCcR61…TXNybU:99:99:bucketthree/nancy:ObjectOnly,RestrictHome

In the example above, joanne and robert each have access to an entire bucket.  Users samuel and nancy each have access to a psuedo-folder within a third bucket.

To have clients choose the bucket, leave off the RestrictHome option.  When RestrictHome is not set, the home directory field is ignored and clients must specify the bucket name as part of their remote path.  With movedat, this just means typing the bucket name in front of the object name.

movedat local.dat s3user@aws.example.com=S3:bucketone/remote.dat

In ExpeDat Desktop, the bucketname will need to be entered into the Remote Prefix field.  Users can type this manually, or you can make it part of an expedat:// URL.  For example, you could create a custom landing page as described belowand modify it to include a bucket prefix.

Changes to svpasswd will be automatically loaded about two minutes after your last modification.

Server Side Encryption

Server Side Encryption (SSE) is enabled by editing /usr/local/expedat/s3handler.sh on the gateway instance.  Look for the #S3KMS= line near the top of the file.  To enable SSE-KMS encryption, remove the leading hash (#) character and replace KMS_ID with the KMS ID of the key you wish to use.  To enable SSE with S3-managed keys (SSE-S3), remove the leading hash (#) character, replace “aws:kms” with “AES256“, and remove the “--API-SSEKMSKeyId=KMS_ID” option.  For example:

S3KMS=”–API-ServerSideEncryption=aws:kms –API-SSEKMSKeyId=KMS_ID”

or

S3KMS=”–API-ServerSideEncryption=AES256″

You can direct users to a specific location in your S3 bucket by creating a gateway link.  The easiest way to do this is to navigate to the location using the ExpeDat Desktop client.  If you want to make a link for downloading a specific file, select that file, then click the Bookmark button.

You will now have an expedat:// link copied into your desktop clipboard.  To create a gateway link, just paste the link onto the end of your gateway URL with a question mark in between.  For example:

http://aws.example.com/?expedat://aws.example.com:8080/Downloadme.dat?u=&h=S3

When a user clicks that link, they will be taken to your gateway page where they receive instructions, an opportunity to download the client software, and a final link to direct the client to download the targeted object.

You can modify the link by pre-filling the username ( u=s3user ) or by prompting the user to upload a file instead of downloading ( &a=s ).  For example:

http://aws.example.com/?expedat://aws.example.com:8080/folder/?u=s3user&h=S3&a=s

See the expedat:// URLs chapter of the ExpeDat Documentation for more details about creating links.

Custom Landing Pages

The CloudDat instance web page from which you download ExpeDat clients is a single PHP file located at /var/www/html/index.php.  The settings at the top of this file allow you to easily customize its look.  You can also rename it to allow other web content on the instance, or make multiple copies with different configurations.

Customization of the instance web page is very similar to that of ExpeDat’s User Links page.  See the User Links documentation for details.

Custom Clients

The clients available for download on the CloudDat instance from AWS Marketplace are pre-configured to show an “S3” object handler as an option in the Handler pop-up menu.  Standard ExpeDat clients are not pre-configured for S3 and must be setup either by the end-user or by embedding a list of handlers.

If you have a standard ExpeDat package, you can embed custom settings into ExpeDat Desktop executables, including a list of Object Handlers.  This list might include the standard “S3” handler as well as any custom handlers you create.  Once you have embedded this list, the Handler pop-up menu will automatically appear when that ExpeDat Desktop executable is run, without the end-user needing to perform any setup.

Instance & EBS Access

Instance system users, such as ec2-user can access the full filesystem of the CloudDat instance, including instance and EBS volumes.  In ExpeDat Desktop, select “None” from the Handler pop-up menu.  In movedat, remove the “=S3” or similar suffix.  System user accounts must have a password assigned to be used with ExpeDat.  For example, “sudo passwd ec2-user".  See the ExpeDat SysAuth documentation for additional information about how system accounts interact with ExpeDat.

Users declared in the AuthFile (/etc/svpasswd) are blocked from accessing the instance filesystem by the ObjectOnlyoption.  If you remove that option from a user's record, then that user will also be able to access the full filesystem with the privileges of the UID and GID specified.  This is not recommended.

If you wish to prevent any users, including system users, from accessing the instance filesystem, add "ObjectOnly" to /etc/servedat.cf.  After modifying servedat.cf, you must signal the server to restart with "sudo killall -HUP servedat".

Glacier

Objects in Amazon Glacier cannot be accessed in real-time, but they can be moved to and from S3.  To upload an object to Glacier, use CloudDat to place the object into S3, then use the AWS APIs or web interface to move it to Glacier.  For automated operations, you can create S3 rules which automatically move new objects to Glacier.  Likewise, use the AWS APIs, web interface, or automation rules to pull objects from Glacier to S3.  Remember that AWS can take several hours to access Glacier objects.

Alternate Endpoints

Some regions, such as China and GovCloud, may require a different S3 endpoint than the default “http://s3.amazonaws.com“.  If you are running CloudDat in a region or VPC which requires a different endpoint, edit /usr/local/expedat/s3handler.sh and add “--region rr-cccc-1 --endpoint http://s3.rr-cccc-1.amazonaws.com/” to the S4CMD command, substituting the appropriate region and endpoint URL.  For example:

S4CMD="python -m s4cmd --region cn-north-1 --endpoint http://s3.cn-north-1.amazonaws.com.cn/"

Web Passwords

This topic requires proficiency with Apache webserver configuration.

You can restrict access to web pages using by sharing the same credentials used by the ExpeDat clients.  The format of the /etc/svpasswd file is compatible with that of an Apache htaccess file.  To restrict the contents of a folder in /var/www/htmlusing svpasswd credentials, create a file in that folder named “.htaccess” with the following contents:

AuthType Basic AuthUserFile /etc/svpasswd AuthName “My S3 Gateway”

For password compatibility, you must use unix-crypt hashes in /etc/svpasswd instead of the default SHA hashes.  To create a unix-crypt hash, run “/usr/local/expedat/mkpasswd -L

Security Warning: If you password-protect your web pages, it is strongly recommended that you install an SSL certificate and configure the webserver to only access authenticated pages using SSL.  Otherwise web browsers will send usernames and passwords in an insecure manner.  ExpeDat clients always encrypt usernames and passwords, but web browsers will only do so if you use SSL.

Custom Handling

This topic requires proficiency with Bash shell scripts.

You can create multiple custom versions of s3handler.sh and access them using different handler names.  Simply make a copy of the /usr/local/expedat/s3handler.sh shell script and modify it as needed.  For example, you might have different S3BUCKET targets or target different SSE options.

After creating a new handler, you must declare it in /etc/servedat.cf.  The following example shows two handlers:

ObjectHandler S3,/usr/local/expedat/s3handler.sh ObjectHandler S3custom,/usr/local/expedat/s3custom.sh

To access the custom handler with movedat, specify its action code on the command line:

movedat local.dat s3user@aws.example.com=S3custom:bucketone/remote.dat

To access the custom handler with ExpeDat Desktop, add its action code in the Opt Setup dialog, then select it in the Handler pop-up menu.  Alternatively, you can follow the instructions above to preconfigure ExpeDat Desktop clients with your custom handlers.

After modifying servedat.cf, you must signal the server to restart with “sudo killall -HUP servedat".

More

Each CloudDat EC2 instance includes a fully functional ExpeDat server.  See the ExpeDat Documentation for more information about customizing it.

Free Trial

Try before you buy

We're so confident in our technology that all of our end-user products are available to try - no strings attached.

Request Trial