Known Issues

The following issues and limitations are known to exist in this version of SyncDat or the systems which it supports.  Also refer to the Limitations section.  For further information about these issues, please contact DEI Technical Support.

Network Attached Storage macOS 10.9 SMB2 Corruption Maximum File Sizes
Memory Limitations Special Characters Maximum Path Length
Windows Command Syntax Non-ASCII Characters Windows Performance
Small MTU Resuming Partial Downloads File Access Conflicts
System Clock Accuracy Windows Symbolic Links Irregular Files
Checkpoint Files "Lost" Transactions Server Restarts
Slow Devices Write Confirmation Hardware Failure
Username Case Sensitivity Filename Case Sensitivity WinSock Reset

Network Attached Storage

Transferring files which are stored on NAS devices may be severely limited by the speed of the legacy protocols used by those devices.  If you must use NAS, see Tech Note 0029 for guidance on tuning NAS devices for high performance.  See the Performance chapter for general guidance on storage performance.

Network attached storage mounts may enforce their own authentication and access controls which may restrict access by the SyncDat server.  See Tech Note 0031 for details.

macOS 10.9 SMB2 File Corruption

macOS version 10.9 may corrupt file data when writing to SMB2 network attached storage.  With 10.9, SMB2 is the default protocol when mounting a Windows file server in the Finder or when using an "smb://" URL.  To avoid this issue, you must mount Windows file servers using the "cifs://" URL, which forces the use of the SMB1 protocol.

Maximum File Sizes

Individual files transferred by SyncDat must be less than 8 exabytes each, or the maximum file size supported by the host operating system.  The total size of all files transfered per client session must be less than 16 exabytes.

Memory Limitations

The maximum number of files which can be synchronized in a single syncdat session is limited by available client system memory (RAM).  This is typically between one and eight million for systems with at least 4 gigabytes of free RAM.

AIX systems may limit available memory, which in turn limits the number of files which may be synchronized.  To increase the available memory for syncdat under AIX, set the LDR_CNTRL environment variable as follows:
  export LDR_CNTRL=MAXDATA=0x8000000

Special Characters

Certain characters should not be used in file and folder names because they have special meaning on some platforms.  When transferring files between different platforms is is possible for a file name which is valid in one context to cause problems in another.  Avoid the following characters in file names:

/   \   :   *   ?     "   '   `   ;   (   )   [   ]   &   |   !   ~   {   }   @   $

For example, if you try to upload a file named "recipes:recent" to a Windows NTFS filesystem, the file or its contents may seem to disappear and servedat may log an error because Windows reserves the ':' character for Alternate Data Streams.

Maximum Path Length

SyncDat limits the total length of a file's pathname to 1024 bytes, including UTF-8 encoding.  Some operating systems, particularly Windows, may impose much shorter limits on pathnames for some operations.  This varies greatly depending on the operating system version, but it is generally safest to keep Windows paths below 256 characters.  Network attached storage devices (NAS/SAN) may also impose shorter limits.

Some Windows systems allow you to exceed the 256 character path limit by prefixing absolute paths with "\\?\" for direct-attached paths or "\\?\UNC\" for UNC paths.  Remember to use forward slashes when specifying paths in a SyncDat client.  Below are some examples:

syncdat -l /home/me/folder syncdat -l /home/me/folder syncdat -l C:\MyFolder\myfile.txt

Note that there are some limitations on using "\\?\" syntax for local paths.  See Windows Command Syntax for details.

As of Windows 10, version 1607, MAX_PATH limitations can be removed by editing the Windows registry or group policy.  ExpeDat and SyncDat applications have adopted Microsoft's mechanism for enabling this on a per-application basis, but as of this writing only the system-wide Windows setting is effective.  When the new long path behavior is enabled, pathnames up to 1024 UTF-8 bytes are supported.

HKLM\SYSTEM\CurrentControlSet\Control\FileSystem LongPathsEnabled 1

For more details about long paths in Windows, see:

Windows Command Syntax

When syncdat is used from a Windows command shell, the following limitations apply.

Arguments, including paths, which contain spaces must be enclosed in double quotes.  For example:

syncdat -l "C:\folder" " Name/folder"

The character sequence \" is ambiguous in Windows.  In most cases, backslash is a path delimiter.  But when it appears immediately before a quote, it is treated as an escape, nullifying the effect of the quote.  This can have the effect of seeming to merge two arguments into one.  For example:

syncdat -l "C:\My Dir\" " Name/My Folder" Local Application: Invalid Argument: Folder transfer requires source and destination directories

To work around this limitation, remove the trailing backslash:

syncdat -l "C:\My Dir" " Name/My Folder"

The 32-bit version of the Windows wsetargv system library contains a bug which prevents local paths starting with "\\?\" from being passed to syncdat.  Therefore you cannot use extended path syntax for local files with the 32-bit version of syncdat.  64-bit Windows does not have this problem, so use the 64-bit version of syncdat if you need to use extended local paths.

Unicode / Non-ASCII Characters

On all platforms, non-ASCII filename characters are encoded as UTF-8 and will be preserved between platforms.

The ability of SyncDat software to display non-ASCII characters will depend upon the fonts and settings of the host system.  This is particularly true for the Windows console.  See the syncdat introduction for steps to improve the display of non-ASCII characters with the Windows console.

In some cases, the Windows version of SyncDat may not correctly display non-ASCII characters even when unicode fonts are available.  However, SyncDat will correctly preserve unicode file name characters even when they are not displayed correctly.

Some systems allow file names to be arbitrary binary strings, which may not be valid unicode.  SyncDat will attempt to preserve such file names, but they are likely to display incorrectly and may cause errors.

If either the source or destination filesystem is a network mount, support for unicode file name characters will depend on the type and configuration of the network mount.

Usernames and passwords should consist of only ASCII letters, numbers, and printable symbols.  The use of other characters, such as extended unicode characters, may work in some environments but is not assured and may compromise security.

Care must be taken to ensure that any shell scripts using syncdat are able to properly handle non-ASCII characters, which may include raw-binary strings.

Windows Performance Limits

Windows has a fixed limit on the amount of memory available for buffering file and network operations.  When the total amount of I/O occurring in the system exceeds this level, operations begin to fail and Windows may become unstable and require a reboot.  SyncDat may report this as:

Insufficient system resources exist to complete the requested service

However, I/O memory is shared by all processes, so its lack may cause errors in any application, process, or device driver.  Errors may be reported in the Windows event log, by other applications you are not using, or they may not be reported at all.

The problem may be triggered by file transfers totaling more than a hundred megabits per second, by multiple applications performing high volume I/O operations, or by the presence of third-party anti-virus, firewall, or virtualization software.  The use of Virtual Machines, Network Attached Storage, Storage Area Networks, or third-party storage drivers increases I/O buffering and may make the problem worse.

Once Windows' paged pool memory has been exhausted, it is necessary to reboot Windows to prevent application crashes and other system-wide problems.  The following Microsoft article explains how to improve paged pool performance, which may allow faster throughput:

If it is not possible to adjust the paged pool, or if doing so does not solve the problem, it may be necessary to disable other software, limit SyncDat's speed, or reconfigure a virtualized environment.

Windows versions prior to Windows 8 and Windows Server 2012 were particularly vulnerable to this problem.  This problem has not been reported with Windows 8, Windows Server 2012, later.

Small MTU

Network links with a Maximum Transmit Unit (MTU) smaller than the common 1500 byte minimum will cause datagrams to be fragmented unless fragmentation has been disabled in the router.  IP fragmentation will magnify packet loss on congested or unreliable networks, resulting in loss rates that are two or three times greater than they would be without fragmentation.  This may cause poor performance with SyncDat.

If IP fragmentation is disabled, then the small MTU will be automatically detected.  If fragmentation cannot be disabled on a link with a small MTU, then use the MaxDatagram setting of servedat, or syncdat to limit the size of SyncDat's datagrams.  Choose a value which is about 100 bytes less than the MTU, or experiment to find the value which achieves the best performance.

Resuming Partial Downloads

If a download is interrupted, it will be restarted from the beginning upon retry.  Interrupted uploads may be resumed from the point of interruption.

File Access Conflicts

The SyncDat server takes steps to ensure that files being uploaded are not written to or read from by other transactions.  However, the mechanisms for this protection are highly dependent upon the operating environment and are not perfect.  In particular, files on network storage devices may be susceptible to modification by other programs while in use.  Users should take care not move or modify files which may be actively transferring.

System Clock Accuracy

Severe problems may occur if the system clock of a host running SyncDat software is not accurate or changes substantially.  See the "System Clocks" section for details.

Windows Symbolic Links

Symbolic Links under Windows are not replicated.  Instead, their targets will be replicated.

In a unix shell environment, the syntax "path1/symlink/../path2/" would access the directory "path2" in the parent directory of the symlink target.  Using such a path via servedat will instead access the directory "path2" in the parent directory of the symlink itself.  In other words, the path would be resolve to "path1/path2/".

Irregular Files

Irregular or "special" files cannot be synchronized.  If an irregular file (device, socket, pipe, etc.) already exists at a destination path, then the synchronization may fail.

Checkpoint Files

The server stores partial uploads using "-sv.tmp" and "-sv.met" files.  See the Upload Status chapter for details.  The syncdat client does not checkpoint downloads: partial downloads must be replaced.  ExpeDat checkpoint files are treated like regular files by the syncdat client.

"Lost" Transactions

If a transaction fails while the server is under heavy load and the network path is experiencing packet loss, a client may receive a "Lost Transaction" error instead of the exact error message.  The exact error message will appear in the server logs.

Server Restarts

Restarting the server via SIGHUP, which is necessary to reload the configuration file, will interrupt any active transactions.  Interrupted transactions will automatically retry if that option has not been disabled..

Slow Devices (CD, DVD, tape, NAS over WAN, etc.)

Targeting files on very slow filesystems or devices such as optical disks, tape drives, network attached storage across a WAN, or any other filesystem with extreme latency may result in severely reduced performance or loss of connectivity.  Targeting slow filesystems on the server may affect the performance and connectivity of all clients accessing that server, even those targeting other filesystems.

Write Confirmation

SyncDat does not consider a file transfer complete until the operating system confirms that all data has been committed to storage.  File transfers may seem to pause for long periods of time at the end if the storage system is slower than the network and the operating system has a large file cache.  The client may report "Waiting for storage" or "Waiting for Server".  When SyncDat reports a final transfer speed, it includes the time it takes to write data to storage.  This may cause SyncDat to seem slower than products which claim to be done before they have finished writing your data.

Hardware & Power Failures

Resuming a synchronization which was interrupted by a hardware or power failure is not recommended.  SyncDat cannot ensure the integrity of storage which has been physically disrupted.  After repairing disrupted hardware, all filesystems and files should be checked for integrity and files which were being written at the time of the disruption should be re-transferred or removed manually.

Username Case Sensitivity

Some authentication databases, notably Active Directory, may treat usernames which differ only by case as the same user.  For example "Joe" and "joe" may match.  Most other databases, including SyncDat's private authentication and system caching, are case sensitive.  To avoid problems and inconsistencies, users should always use the correct capitalization when typing their username.

Filename Case Sensitivity

Some file systems, notably Windows and macOS native filesystems, may treat filenames which differ only by case as the same file.  For example "Data" and "data" may match.  Other filesystems, including UFS and EXT, are case sensitive.  Network Attached Storage protocols may introduce additional complications.  To avoid problems and possible data loss, always use correct capitalization when transferring files and never permit two files whose names differ only by case.

WinSock Reset

If a Windows application improperly accesses the WinSock network layer, it can disrupt networking for all applications, including SyncDat.  The following error messages indicate that a WinSock reset has occurred:

NetIn Unable to receive from network NetOut Unable to send to network

If you see these errors, restart the SyncDat application and check the Windows system event logs for warnings about WinSock.  Disable unneeded network services.  For example, the "WinHTTP Web Proxy Audio-Discovery Service" is known to cause this problem.