Hi All,
Have been a SyncBackSE user for years and generally have found the application awesome.
Recently I switched to Mozilla Firefox, and have found my backups have begun to fail.
After tracing through debug logs, running wireshark sessions, and other troubleshooting; I have come to the following conclusion:
Facts:
- The Mozilla Firefox profile cache directory has ~70,000 files in it (%AppData%\Local\Mozilla\Firefox\Profiles\*ProfileID*\cache2\entries)
- The directory is very active - contents change frequently. New files added. Old files deleted.
- A file-listing over SMB of this directory takes ~25 seconds to complete.
- SyncBackSE is running in mirror source->destination mode
- The backup profile is configured to use 'Standard Windows file copying' (although I have tried 'Windows Explorer method of file copying' too)
- COPYING each file from source to destination takes a reasonable amount of time
Issue:
- DELETING each file on the destination which no longer exist on the source takes ~25 seconds!
- This is multiplied by EACH FILE which must be deleted on the destination!
Reason:
- Wireshark investigation reveals that whichever method SyncBackSE is using to issue a DELETE on a single file, it first issues an un-necessary full directory file listing, incurring the ~25 second wait time to retrieve the list of ~70,000 files.
Comparison:
- Mapping the SMB path in question to a drive-letter and then using the command prompt to DEL one of the same files incurs the same behavior pattern - it appears the command line DEL command also issues an un-necessary full directory file-listing for each single file deleted
- Using C# to issue a System.IO.File.Delete() on the same files does NOT incur the same pattern. A full directory file-listing is NOT done, and the file is deleted instantly.
Questions:
- Has anyone else had this issue?
- SyncBack support: Is this expected behavior when set to 'Standard Windows file copying' ? I would expect it to happen if set to 'Windows Explorer method of file copying', but I would expect 'Standard Windows file copying' to use similar kernel calls as C# uses and not incur the penalty of a full directory file-listing when deleting a specific, individual file.
Addendum:
- I am not able to change to another copying method than SMB
- The firefox cache may not be important - some may say skip it - but that is not the only directory exhibiting this behavior; it is only a good example
- It is not sufficient to expect the server to return the file listing faster. This is simply not possible on all SMB implementations; and should not be necessary for deleting a single specific file
Have been a SyncBackSE user for years and generally have found the application awesome.
Recently I switched to Mozilla Firefox, and have found my backups have begun to fail.
After tracing through debug logs, running wireshark sessions, and other troubleshooting; I have come to the following conclusion:
Facts:
- The Mozilla Firefox profile cache directory has ~70,000 files in it (%AppData%\Local\Mozilla\Firefox\Profiles\*ProfileID*\cache2\entries)
- The directory is very active - contents change frequently. New files added. Old files deleted.
- A file-listing over SMB of this directory takes ~25 seconds to complete.
- SyncBackSE is running in mirror source->destination mode
- The backup profile is configured to use 'Standard Windows file copying' (although I have tried 'Windows Explorer method of file copying' too)
- COPYING each file from source to destination takes a reasonable amount of time
Issue:
- DELETING each file on the destination which no longer exist on the source takes ~25 seconds!
- This is multiplied by EACH FILE which must be deleted on the destination!
Reason:
- Wireshark investigation reveals that whichever method SyncBackSE is using to issue a DELETE on a single file, it first issues an un-necessary full directory file listing, incurring the ~25 second wait time to retrieve the list of ~70,000 files.
Comparison:
- Mapping the SMB path in question to a drive-letter and then using the command prompt to DEL one of the same files incurs the same behavior pattern - it appears the command line DEL command also issues an un-necessary full directory file-listing for each single file deleted
- Using C# to issue a System.IO.File.Delete() on the same files does NOT incur the same pattern. A full directory file-listing is NOT done, and the file is deleted instantly.
Questions:
- Has anyone else had this issue?
- SyncBack support: Is this expected behavior when set to 'Standard Windows file copying' ? I would expect it to happen if set to 'Windows Explorer method of file copying', but I would expect 'Standard Windows file copying' to use similar kernel calls as C# uses and not incur the penalty of a full directory file-listing when deleting a specific, individual file.
Addendum:
- I am not able to change to another copying method than SMB
- The firefox cache may not be important - some may say skip it - but that is not the only directory exhibiting this behavior; it is only a good example
- It is not sufficient to expect the server to return the file listing faster. This is simply not possible on all SMB implementations; and should not be necessary for deleting a single specific file
Statistics: Posted by jmorgannz — Wed Jan 23, 2019 4:06 am