Amazon Web Services
Each AWS instance of CloudDat includes a fully customizable ExpeDat server. ExpeDat is DEI’s high performance file transfer software which uses our proprietary transport protocol to move data across the Internet, overcoming distance, latency, and congestion.
With a standard install, CloudDat for AWS provides access to a single S3 bucket for a single username. Additional usernames, multiple buckets, access to EBS and instance storage, landing page customization, and full ExpeDat functionality can be enabled by modifying the CloudDat configurations.
A standard install of the CloudDat on AWS uses the following ExpeDat configuration options.
/etc/servedat.cfties the “S3” action code supplied by clients to the
s3handler.shshell script on the CloudDat EC2 instance. See the Object Handlers documentation.
/etc/svpasswd, which resembles a unix password file. The
s3usercreated during initial setup is given the
ObjectOnlyrestriction, which means it cannot access the instance filesystem. See the ExpeDat AuthFile documentation. If you customize this file, be sure to answer ‘n’ if you re-run
s3setup.shwhen it asks about replacing existing credentials.
/etc/servedat.cfallows system accounts on the CloudDat instance, such as
ec2-user, to access both S3 and the instance filesystem. You must assign a password to a system account for it to be useable with ExpeDat. For example, “sudo passwd ec2-user". See the ExpeDat SysAuth documentation.
Following are some of the ways you can modify these settings to customize CloudDat.
New versions of CloudDat for perpetually licensed software can be downloaded directly from your DEI Customer Site. New versions of CloudDat from AWS Marketplace are obtained by launching a new CloudDat instance from AWS Marketplace and copying over configuration and other customer data from the old instance to the new one.
If you have performed any of the advanced customizations on this page, you will need to preserve the following files:
General advice for updating DEI software can be found in Tech Note 0012.
The initial installation creates a single S3 user. To create additional credentials, edit
/etc/svpasswd, using that first entry as a template. For example, after adding a second user,
svpasswd might look like:
The second field of
svpasswd can contain a plain text password, but it is safer to generate an SHA hash using the
/usr/local/expedat/mkpasswd utility. After modifying
svpasswd, the server will automatically load the new credentials in about 2 minutes.
With multiple user credentials, you can apply different restrictions and configurations to different accounts, as discussed below.
The final field of an
svpasswd) record contains a list of access restrictions. Commonly used restrictions are
ReadOnly (no uploads or changes allowed) and
WriteListOnly (list, delete, rename, upload, but no download). The
ObjectOnly option prevents that user from accessing the instance filesystem. The
RestrictHome option can be used to restrict access within a bucket, as described next.
See the ExpeDat AuthFile documentation for more options.
The default CloudDat user account, shown above, has access to the entire S3 bucket. You can restrict gateway users to just a portion of the bucket by assigning a
RestrictHome prefix. For example,
This will insert the prefix “
webfolder/” in front of all paths accessed by ExpeDat clients using the “
s3user” name and will not allow them to access any other objects. It is like restricting the user to a single folder. You can create multiple users, each restricted to a different prefix. Changes to
svpasswd will be automatically loaded about two minutes after your last modification.
Step 1 of the Install process asks you to choose an S3 bucket to access. Setting a fixed name allows clients to access S3 storage without knowing the bucket name. If you leave this blank, clients must specify a bucket name as the first part of the remote path. This allows you to access multiple buckets through a single CloudDat instance.
If you’ve already setup the gateway, the easiest way to modify the bucket name is by editing
/usr/local/expedat/s3handler.sh to change the “
S3BUCKET=” setting at the top.
S3BUCKET is blank, the bucket must be specified either by the client or as part of the
svpasswd home prefix. If you follow the Multiple Users and Segmented Buckets instructions above, you can have different users accessing different buckets all from the same CloudDat instance.
joanne:frYypg…0HCxHA:99:99:bucketone:ObjectOnly,RestrictHome robert:SddKMR…jSHh7g:99:99:buckettwo:ObjectOnly,RestrictHome samuel:Yd3751…PWFgIQ:99:99:bucketthree/sam:ObjectOnly,RestrictHome nancy:nCcR61…TXNybU:99:99:bucketthree/nancy:ObjectOnly,RestrictHome
In the example above,
robert each have access to an entire bucket. Users
nancy each have access to a psuedo-folder within a third bucket.
To have clients choose the bucket, leave off the
RestrictHome option. When
RestrictHome is not set, the home directory field is ignored and clients must specify the bucket name as part of their remote path. With
movedat, this just means typing the bucket name in front of the object name.
movedat local.dat email@example.com=S3:bucketone/remote.dat
ExpeDat Desktop, the bucketname will need to be entered into the
Remote Prefix field. Users can type this manually, or you can make it part of an expedat:// URL. For example, you could create a custom landing page as described belowand modify it to include a bucket prefix.
svpasswd will be automatically loaded about two minutes after your last modification.
Server Side Encryption (SSE) is enabled by editing
/usr/local/expedat/s3handler.sh on the gateway instance. Look for the
#S3KMS= line near the top of the file. To enable SSE-KMS encryption, remove the leading hash (
#) character and replace
KMS_ID with the KMS ID of the key you wish to use. To enable SSE with S3-managed keys (SSE-S3), remove the leading hash (
#) character, replace “
aws:kms” with “
AES256“, and remove the “
--API-SSEKMSKeyId=KMS_ID” option. For example:
You can direct users to a specific location in your S3 bucket by creating a gateway link. The easiest way to do this is to navigate to the location using the
ExpeDat Desktop client. If you want to make a link for downloading a specific file, select that file, then click the Bookmark button.
You will now have an
expedat:// link copied into your desktop clipboard. To create a gateway link, just paste the link onto the end of your gateway URL with a question mark in between. For example:
When a user clicks that link, they will be taken to your gateway page where they receive instructions, an opportunity to download the client software, and a final link to direct the client to download the targeted object.
You can modify the link by pre-filling the username (
u=s3user ) or by prompting the user to upload a file instead of downloading (
&a=s ). For example:
See the expedat:// URLs chapter of the ExpeDat Documentation for more details about creating links.
The CloudDat instance web page from which you download ExpeDat clients is a single PHP file located at
/var/www/html/index.php. The settings at the top of this file allow you to easily customize its look. You can also rename it to allow other web content on the instance, or make multiple copies with different configurations.
Customization of the instance web page is very similar to that of ExpeDat’s User Links page. See the User Links documentation for details.
The clients available for download on the CloudDat instance from AWS Marketplace are pre-configured to show an “S3” object handler as an option in the Handler pop-up menu. Standard ExpeDat clients are not pre-configured for S3 and must be setup either by the end-user or by embedding a list of handlers.
If you have a standard ExpeDat package, you can embed custom settings into
ExpeDat Desktop executables, including a list of Object Handlers. This list might include the standard “S3” handler as well as any custom handlers you create. Once you have embedded this list, the Handler pop-up menu will automatically appear when that
ExpeDat Desktop executable is run, without the end-user needing to perform any setup.
Instance system users, such as
ec2-user can access the full filesystem of the CloudDat instance, including instance and EBS volumes. In
ExpeDat Desktop, select “None” from the Handler pop-up menu. In
movedat, remove the “
=S3” or similar suffix. System user accounts must have a password assigned to be used with ExpeDat. For example, “sudo passwd ec2-user". See the ExpeDat SysAuth documentation for additional information about how system accounts interact with ExpeDat.
Users declared in the
/etc/svpasswd) are blocked from accessing the instance filesystem by the
ObjectOnly option. If you remove that option from a user's record, then that user will also be able to access the full filesystem with the privileges of the UID and GID specified. This is not recommended without additional restrictions such as RestrictHome and AllowPath to control which paths are visible to the user.
If you wish to prevent any users, including system users, from accessing the instance filesystem, add "
/etc/servedat.cf. After modifying
servedat.cf, you must signal the server to restart with "sudo killall -HUP servedat".
Objects in Amazon Glacier cannot be accessed in real-time, but they can be moved to and from S3. To upload an object to Glacier, use CloudDat to place the object into S3, then use the AWS APIs or web interface to move it to Glacier. For automated operations, you can create S3 rules which automatically move new objects to Glacier. Likewise, use the AWS APIs, web interface, or automation rules to pull objects from Glacier to S3. Remember that AWS can take several hours to access Glacier objects.
Some regions, such as China and GovCloud, may require a different S3 endpoint than the default “
http://s3.amazonaws.com“. If you are running CloudDat in a region or VPC which requires a different endpoint, edit
/usr/local/expedat/s3handler.sh and add “
--region rr-cccc-1 --endpoint http://s3.rr-cccc-1.amazonaws.com/” to the
S4CMD command, substituting the appropriate region and endpoint URL. For example:
S4CMD="python -m s4cmd --region cn-north-1 --endpoint http://s3.cn-north-1.amazonaws.com.cn/"
This topic requires proficiency with Apache webserver configuration.
You can restrict access to web pages using by sharing the same credentials used by the ExpeDat clients. The format of the
/etc/svpasswd file is compatible with that of an Apache
htaccess file. To restrict the contents of a folder in
svpasswd credentials, create a file in that folder named “
.htaccess” with the following contents:
AuthType Basic AuthUserFile /etc/svpasswd AuthName “My S3 Gateway”
For password compatibility, you must use unix-crypt hashes in
/etc/svpasswd instead of the default SHA hashes. To create a unix-crypt hash, run “
Security Warning: If you password-protect your web pages, it is strongly recommended that you install an SSL certificate and configure the webserver to only access authenticated pages using SSL. Otherwise web browsers will send usernames and passwords in an insecure manner. ExpeDat clients always encrypt usernames and passwords, but web browsers will only do so if you use SSL.
This topic requires proficiency with Bash shell scripts.
You can create multiple custom versions of
s3handler.sh and access them using different handler names. Simply make a copy of the
/usr/local/expedat/s3handler.sh shell script and modify it as needed. For example, you might have different
S3BUCKET targets or target different SSE options.
After creating a new handler, you must declare it in
/etc/servedat.cf. The following example shows two handlers:
ObjectHandler S3,/usr/local/expedat/s3handler.sh ObjectHandler S3custom,/usr/local/expedat/s3custom.sh
To access the custom handler with
movedat, specify its action code on the command line:
movedat local.dat firstname.lastname@example.org=S3custom:bucketone/remote.dat
To access the custom handler with
ExpeDat Desktop, add its action code in the Opt Setup dialog, then select it in the Handler pop-up menu. Alternatively, you can follow the instructions above to preconfigure
ExpeDat Desktop clients with your custom handlers.
servedat.cf, you must signal the server to restart with “sudo killall -HUP servedat".