Known Issues

The following issues and limitations are known to exist in this version of ExpeDat or the systems which it supports.  For further information about these issues, please contact DEI Technical Support.

Network Attached Storage OS X 10.9 SMB2 Corruption Maximum File Sizes
Memory Limitations Special Characters Maximum Path Length
Windows Command Syntax Non-ASCII Characters Windows Performance
Small MTU Compressed Upload Resume Drag and Drop
Listing Performance File Access Conflicts Compression Performance
Compression Statistics Streaming Statistics Resume Statistics
System Clock Accuracy OS Progress Symbolic Links
Irregular Files Checkpoint Files "Lost" Transactions
Server Restarts Mass Deletion Hierarchical Wild-Cards
Windows Browser Sorting Slow Devices Write Confirmation
Hardware Failure Username Case Sensitivity Filename Case Sensitivity
WinSock Reset

Network Attached Storage

Transferring files which are stored on NAS devices may be severely limited by the speed of the legacy protocols used by those devices.  If you must use NAS, see Tech Note 0029 for guidance on tuning NAS devices for high performance.  See the Performance chapter for general guidance on storage performance.

Network attached storage mounts may enforce their own authentication and access controls which may restrict access by the ExpeDat server.  See Tech Note 0031 for details.

Mac OS X 10.9 SMB2 File Corruption

Mac OS X version 10.9 may corrupt file data when writing to SMB2 network attached storage.  With 10.9, SMB2 is the default protocol when mounting a Windows file server in the Finder or when using an "smb://" URL.  To avoid this issue, you must mount Windows file servers using the "cifs://" URL, which forces the use of the SMB1 protocol.

Maximum File Sizes

Individual files transferred by ExpeDat must be less than 8 exabytes each, or the maximum file size supported by the host operating system.  The total size of all files transfered per client session must be less than 16 exabytes.

Memory Limitations

The maximum number of individual files which can be transferred one-at-a-time (as opposed to Streaming) is limited by available client system memory (RAM).  This is typically about one million files for MTPexpedat or DropDat, and about eight million for movedat.  When transferring files with Streaming Folders, the limit is approximately 2 billion files, regardless of available memory.

AIX systems may limit available memory, which in turn limits the number of files which may be transferred one-at-a-time.  To increase the available memory for movedat under AIX, set the LDR_CNTRL environment variable as follows:
  export LDR_CNTRL=MAXDATA=0x8000000

Special Characters

Certain characters should not be used in file and folder names because they have special meaning on some platforms.  When transferring files between different platforms is is possible for a file name which is valid in one context to cause problems in another.  Avoid the following characters in file names:

/   \   :   *   ?     "   '   `   ;   (   )   [   ]   &   |   !   ~   {   }   @   $

It is also best to avoid spaces in file and folder names, particularly when using Packaging or Object Handlers.

For example, if you try to upload a file named "recipes:recent" to a Windows NTFS filesystem, the file or its contents may seem to disappear and servedat may log an error because Windows reserves the ':' character for Alternate Data Streams.

Maximum Path Length

ExpeDat limits the total length of a file's pathname to 1024 bytes, including UTF-8 encoding.  Some operating systems, particularly Windows, may impose much shorter limits on pathnames for some operations.  This varies greatly depending on the operating system version, but it is generally safest to keep Windows paths below 256 characters.  Network attached storage devices (NAS/SAN) may also impose shorter limits.

Some Windows systems allow you to exceed the 256 character path limit by prefixing absolute paths with "\\?\" for direct-attached paths or "\\?\UNC\" for UNC paths.  Remember to use forward slashes when specifying paths in an ExpeDat client.  Below are some examples using movedat:

movedat user@server.example.com://?/C:/Users/Administrator/ movedat user@server.example.com://?/UNC/nas_server/folder/ movedat localfile user@server.example.com://?/UNC/nas_server/folder/

Note that there are some limitations on using "\\?\" syntax for local paths.  See Windows Command Syntax for details.

For more details about long paths in Windows, see:
http://msdn.microsoft.com/en-us/library/windows/desktop/aa365247(v=vs.85).aspx#maxpath

Windows Command Syntax

When movedat is used from a Windows command shell, the following limitations apply.

Arguments, including paths, which contain spaces must be enclosed in double quotes.  For example:

movedat "user@server.example.com://?/C:/Users/My Name/file.txt" "C:\tmp"

The character sequence \" is ambiguous in Windows.  In most cases, backslash is a path delimiter.  But when it appears immediately before a quote, it is treated as an escape, nullifying the effect of the quote.  This can have the effect of seeming to merge two arguments into one.  For example:

movedat -h "C:\My Dir\" "user@server.example.com://?/C:/Users/My Name/My Folder" Local Application: Invalid Argument: Folder transfer requires source and destination directories

To work around this limitation, remove the trailing backslash:

movedat -h "C:\My Dir" "user@server.example.com://?/C:/Users/My Name/My Folder"

The 32-bit version of the Windows wsetargv library contains a bug which prevents local paths starting with "\\?\" from being passed to movedat.  Therefore you cannot use extended path syntax for local files with the 32-bit version of movedat.  64-bit Windows does not have this problem, so use the 64-bit version of movedat if you need to use extended local paths.

Unicode / Non-ASCII Characters

On all platforms, non-ASCII filename characters are encoded as UTF-8 and will be preserved between platforms.

The ability of ExpeDat software to display non-ASCII characters will depend upon the fonts and settings of the host system.  This is particularly true for the Windows console.  See the movedat introduction for steps to improve the display of non-ASCII characters with the Windows console.

In some cases, the Windows version of ExpeDat may not correctly display non-ASCII characters even when unicode fonts are available.  However, ExpeDat will correctly preserve unicode file name characters even when they are not displayed correctly.

Some systems allow file names to be arbitrary binary strings, which may not be valid unicode.  ExpeDat will attempt to preserve such file names, but they are likely to display incorrectly and may cause errors.

If either the source or destination filesystem is a network mount, support for unicode file name characters will depend on the type and configuration of the network mount.

Usernames and passwords should consist of only ASCII letters, numbers, and printable symbols.  The use of other characters, such as extended unicode characters, may work in some environments but is not assured and may compromise security.

Care must be taken to ensure that any shell scripts using movedat, Object Handlers, or Packaging scripts are able to properly handle non-ASCII characters, which may include raw-binary strings.

Windows Performance Limits

Windows has a fixed limit on the amount of memory available for buffering file and network operations.  When the total amount of I/O occurring in the system exceeds this level, operations begin to fail and Windows may become unstable and require a reboot.  ExpeDat may report this as:

Insufficient system resources exist to complete the requested service

However, I/O memory is shared by all processes, so its lack may cause errors in any application, process, or device driver.  Errors may be reported in the Windows event log, by other applications you are not using, or they may not be reported at all.

The problem may be triggered by file transfers totaling more than a hundred megabits per second, by multiple applications performing high volume I/O operations, or by the presence of third-party anti-virus, firewall, or virtualization software.  The use of Virtual Machines, Network Attached Storage, Storage Area Networks, or third-party storage drivers increases I/O buffering and may make the problem worse.

Once Windows' paged pool memory has been exhausted, it is necessary to reboot Windows to prevent application crashes and other system-wide problems.  The following Microsoft article explains how to improve paged pool performance, which may allow faster throughput:

http://support.microsoft.com/kb/312362

If it is not possible to adjust the paged pool, or if doing so does not solve the problem, it may be necessary to disable other software, limit ExpeDat's speed, or reconfigure a virtualized environment.

Windows XP and Server 2003 are particularly vulnerable to this problem, but it has also been reported with Windows Vista and Windows Server 2008.  This problem has not been reported with Windows Server 2012.

Small MTU

Network links with a Maximum Transmit Unit (MTU) smaller than the common 1500 byte minimum will cause datagrams to be fragmented unless fragmentation has been disabled in the router.  IP fragmentation will magnify packet loss on congested or unreliable networks, resulting in loss rates that are two or three times greater than they would be without fragmentation.  This may cause poor performance with ExpeDat.

If IP fragmentation is disabled, then the small MTU will be automatically detected.  If fragmentation cannot be disabled on a link with a small MTU, then use the MaxDatagram setting of servedat, MTPexpedat, or movedat to limit the size of ExpeDat's datagrams.  Choose a value which is about 100 bytes less than the MTU, or experiment to find the value which achieves the best performance.

Resuming an Upload with Compression

When resuming an interrupted upload, compression cannot be used.  To resume an upload, either compression must be disabled or the upload must be restarted.  This will be done automatically if the client is non-interactive.

Drag and Drop

The MTPexpedat GUI does not currently support dragging remote files to a local destination.  Instead, users can drag a local folder into the local browser pane to select that folder as the destination.

Dragging local files to the MTPexpedat GUI will select those files in the local file browser.  If the dragged files are contained in multiple folders, only the those in one of the folders will be displayed.

Listing Performance

Listing a very large directory, either locally or remotely, may temporarily reduce the performance of any file transfers occurring at the same time.

File Access Conflicts

The ExpeDat server takes steps to ensure that files being uploaded are not written to or read from by other transactions.  However, the mechanisms for this protection are highly dependent upon the operating environment and are not perfect.  In particular, files on network storage devices may be susceptible to modification by other programs while in use.  Users should take care not move or modify files which may be actively transferring.

Compression Performance

Compression can negatively affect performance, particularly if the data being transferred is encrypted or already compressed.  Compression should only be used if the data being transferred is known to be compressible.  See Tech Note 0014 for more details about compression.

Compression Statistics

Information about the progress and speed of a file transfer may not be accurate when compression is being used.  The server will log the amount of data actually transferred across the network (after compression).  If a compressed transfer is interrupted, the amount received may not match the amount sent due to buffering.

Streaming Folders Statistics

Information about the progress and speed of a file transfer may not be accurate when Streaming Folders is being used.  The current progress may be displayed, but the total size of the folder is not known until it has been completely transmitted.  If a Streaming Folders transfer is interrupted, the files actually received may not match the files reported sent due to buffering.

Resume Statistics

After resuming an interrupted transfer, clients may report amounts of data or speeds which include data transferred in a previous session.  The server logs only count the data moved during the current session.

System Clock Accuracy

Various problems may occur if the system clock of a host running ExpeDat software is not accurate or changes substantially.  Examples include interrupted transfers, failure to resume interrupted transfers, incorrect file time stamps, and poor performance.  All ExpeDat hosts, especially servers, should enable time synchronization software and ensure that the system clock remains accurate.  You can test whether the clocks of a client and server are synchronized by using mtping to query the server from the client and observing the "Clock Skew".  A Clock Skew of more than a few of seconds should be corrected.

Operating System Under-Reports Progress

During file transfer, some operating environments may report the size of the arriving file as being much smaller than it actually is.  This is most notable with Windows Server 2008 running in an EC2 instance, but may occur on other systems as well.  Once the transfer has stopped, the operating system should show the correct size of the file.

Symbolic Links

Accessing files and folders via symbolic links may produce unexpected results.  In order to access the target of a symbolic link, the permissions of the link's parent folder, the link itself, the target's parent folder, and the target itself all must permit access.

Attempting to download a link which points to a folder may result in an error or a zero length file.  To list a folder pointed to by a symbolic link in movedat, place an explicit "/" character after the link name.  In MTPexpedat, symbolic links which point to a folder may be displayed with a folder icon and a slash after the name.  You may double click on these to browser the target folder.  In some cases, such as with older ExpeDat servers, symbolic links to folders may look like regular files.  To list a folder pointed to by an untyped symbolic link in MTPexpedat, enter its path in the Remote Path field and click the remote Refresh refresh button.

In a unix shell environment, the syntax "path1/symlink/../path2/" would access the folder "path2" in the parent folder of the symlink target.  Using such a path via servedat will instead access the folder "path2" in the parent folder of the symlink itself.  In other words, the path would be resolve to "path1/path2/".

With the Mac OS X version of MTPexpedat, browsing into a local folder pointed to by a symbolic link may change the Local Path to that of the target.  In that case, clicking Parent may take you to the parent of the target, not the folder you were just in.

Irregular Files

Irregular or "special" files cannot be targeted as the destination of a Send.  If an irregular file (device, socket, pipe, etc.) already exists at the destination path, then the upload will fail with an "Object Unavailable" error.  See the movedat and MTPexpedat documentation for details.

Irregular or "special" files such as devices, sockets, pipes, and Windows symbolic links, cannot be copied.  If these files are included in a directory or folder being transferred, they may be ignored or they may cause the transfer to fail.

Checkpoint Files

Different formats are used to store information about partial transfers.

The server stores partial uploads using "-sv.tmp" and "-sv.met" files.  See the Upload Status chapter for details.

The Windows clients store partial downloads using ".TMP" and ".CHK" files.  All other clients store partial downloads using a suffix of the form "#00000000.TMP".  If the NoCheck option is enabled, the movedat client will download directly to a target file.

"Lost" Transactions

If a transaction fails while the server is under heavy load and the network path is experiencing packet loss, a client may receive a "Lost Transaction" error instead of the exact error message.  The exact error message will appear in the server logs.

Server Restarts

Restarting the server via SIGHUP, which is necessary to reload the configuration file, will interrupt any active transactions.  Interrupted transactions will automatically retry for MTPexpedat, DropDat, and for movedat if it has not been disabled.

Mass Deletion

Folders can only be deleted if they are empty.  Mass deletion of a folder's contents is not yet supported.

Hierarchical Wild-Cards

Remote directories selected using the movedat wild-card option are not currently subject to hierarchical processing.  To transfer the contents of such remote directories, they must be explicitly named on the command line.

MTPexpedat Windows Sorting

In the Windows version of MTPexpedat, the browser sort order reverts to Name-Ascending whenever a new listing is performed.

Slow Devices (CD, DVD, tape, NAS over WAN, etc.)

Targeting files on very slow filesystems or devices such as optical disks, tape drives, network attached storage across a WAN, or any other filesystem with extreme latency may result in severely reduced performance or loss of connectivity.  Targeting slow filesystems on the server may affect the performance and connectivity of all clients accessing that server, even those targeting other filesystems.

Write Confirmation

ExpeDat does not consider a file transfer complete until the operating system confirms that all data has been committed to storage.  File transfers may seem to pause for long periods of time at the end if the storage system is slower than the network and the operating system has a large file cache.  The client may report "Waiting for storage" or "Waiting for Server".  When ExpeDat reports the final transfer speed, it includes the time it takes to write data to storage.  This may cause ExpeDat to seem slower than products which claim to be done before they have finished writing your data.

Hardware & Power Failures

Resuming a transfer which was interrupted by a hardware or power failure is not recommended.  ExpeDat cannot ensure the integrity of storage which has been physically disrupted.  After repairing disrupted hardware, all filesystems and files should be checked for integrity and files which were being written at the time of the disruption should be re-transferred from the beginning.

Username Case Sensitivity

Some authentication databases, notably Active Directory, may treat usernames which differ only by case as the same user.  For example "Joe" and "joe" may match.  Most other databases, including ExpeDat's private authentication and system caching, are case sensitive.  To avoid problems and inconsistencies, users should always use the correct capitalization when typing their username.

Filename Case Sensitivity

Some file systems, notably Windows and Mac native filesystems, may treat filenames which differ only by case as the same file.  For example "Data" and "data" may match.  Other filesystems, including UFS and EXT, are case sensitive.  Network Attached Storage protocols may introduce additional complications.  To avoid problems and possible data loss, always use correct capitalization when transferring files and never permit two files whose names differ only by case.

WinSock Reset

If a Windows application improperly accesses the WinSock network layer, it can disrupt networking for all applications, including ExpeDat.  The following error messages indicate that a WinSock reset has occurred:

NetIn Unable to receive from network NetOut Unable to send to network

If you see these errors, restart the ExpeDat application and check the Windows system event logs for warnings about WinSock.  Disable unneeded network services.  For example, the "WinHTTP Web Proxy Audio-Discovery Service" is known to cause this problem.