overwrite in loop

[BruceLee]

Hi,
 
I'm currently testing the enterprise edition.
I want to synchronize files from a FTP server down to a local drive.
Since it's a one way sync with delete it needs to check if the file is already existing locally.
that works well....but
it keeps resyncing the files again and again. Minutes, hours and datestamp are correct.
Simple ftp transfer in the internal network.
The favorite settings are set to "enable" in "keep modify time" settings.
Remote FTP server is an IIS 8.5 with the following supported commands:
 ABOR
 ACCT
 ADAT *
 ALLO
 APPE
 AUTH
 CCC
 CDUP
 CWD
 DELE
 ENC *
 EPRT
 EPSV
 FEAT
 HELP
 HOST
 LANG
 LIST
 MDTM
 MIC *
 MKD
 MODE
 NLST
 NOOP
 OPTS
 PASS
 PASV
 PBSZ
 PORT
 PROT
 PWD
 QUIT
 REIN
 REST
 RETR
 RMD
 RNFR
 RNTO
 SITE
 SIZE
 SMNT
 STAT
 STOR
 STOU
 STRU
 SYST
 TYPE
 USER
 XCUP
 XCWD
 XMKD
 XPWD
 XRMD
 
 
This is the first sync before the file exists locally.
2016-11-08T09:49:51Z Operation begin
2016-11-08T09:49:51Z Copying file. Source="::{82AA9188-44E0-40B9-B956-43A10C315B4F}\::{6DDED4A4-3958-49C1-8E87-D9CE657B977A}/LOGS/20160921 231303 01 {298B0B39-D344-45DC-87F3-F33382EEE806}.log", Destination="E:\LOGS\20160921 231303 01 {298B0B39-D344-45DC-87F3-F33382EEE806}.log"
2016-11-08T09:49:51Z Getting properties (Size, Date modified) of "::{82AA9188-44E0-40B9-B956-43A10C315B4F}\::{6DDED4A4-3958-49C1-8E87-D9CE657B977A}/LOGS/20160921 231303 01 {298B0B39-D344-45DC-87F3-F33382EEE806}.log"
2016-11-08T09:49:51Z Item: Size=9506, Date modified=2016-09-21T21:14:00
2016-11-08T09:49:51Z Transfer restarting at position 0.
2016-11-08T09:49:51Z 9506 bytes transferred. (97,7 KB/s) (95 ms)
2016-11-08T09:49:51Z Getting properties (Date modified, Date created) of "::{82AA9188-44E0-40B9-B956-43A10C315B4F}\::{6DDED4A4-3958-49C1-8E87-D9CE657B977A}/LOGS/20160921 231303 01 {298B0B39-D344-45DC-87F3-F33382EEE806}.log"
2016-11-08T09:49:51Z Item: Date modified=2016-09-21T21:14:00
2016-11-08T09:49:51Z Setting properties (Date modified) of "E:\LOGS\20160921 231303 01 {298B0B39-D344-45DC-87F3-F33382EEE806}.log"
2016-11-08T09:49:51Z Getting properties (Size) of "::{82AA9188-44E0-40B9-B956-43A10C315B4F}\::{6DDED4A4-3958-49C1-8E87-D9CE657B977A}/LOGS/20160921 231303 01 {298B0B39-D344-45DC-87F3-F33382EEE806}.log"
2016-11-08T09:49:51Z Item: Size=9506
2016-11-08T09:49:51Z Using cached properties
2016-11-08T09:49:51Z Getting properties (Size) of "E:\LOGS\20160921 231303 01 {298B0B39-D344-45DC-87F3-F33382EEE806}.log"
2016-11-08T09:49:51Z Item: Size=9506
2016-11-08T09:49:51Z Source File Size=9506, Destination file size=9506
2016-11-08T09:49:51Z Operation end

And this is the second that should skip the file (default ruleset of smartftp is in use with checked setting of hash values too):
2016-11-08T09:49:57Z Operation begin
2016-11-08T09:49:57Z Copying file. Source="::{82AA9188-44E0-40B9-B956-43A10C315B4F}\::{6DDED4A4-3958-49C1-8E87-D9CE657B977A}/LOGS/20160921 231303 01 {298B0B39-D344-45DC-87F3-F33382EEE806}.log", Destination="E:\LOGS\20160921 231303 01 {298B0B39-D344-45DC-87F3-F33382EEE806}.log"
2016-11-08T09:49:57Z Getting properties (Size, Date modified) of "::{82AA9188-44E0-40B9-B956-43A10C315B4F}\::{6DDED4A4-3958-49C1-8E87-D9CE657B977A}/LOGS/20160921 231303 01 {298B0B39-D344-45DC-87F3-F33382EEE806}.log"
2016-11-08T09:49:57Z Item: Size=9506, Date modified=2016-09-21T21:14:00
2016-11-08T09:49:57Z Getting properties (Size, Date modified) of "E:\LOGS\20160921 231303 01 {298B0B39-D344-45DC-87F3-F33382EEE806}.log"
2016-11-08T09:49:57Z Item: Size=9506, Date modified=2016-09-21T21:14:00
2016-11-08T09:49:57Z Source file: Size=9506, Date modified=2016-09-21T21:14:00
2016-11-08T09:49:57Z Destination file: Size=9506, Date modified=2016-09-21T21:14:00
2016-11-08T09:49:57Z No rule matched. Default action="Overwrite".
2016-11-08T09:49:57Z Transfer restarting at position 0.
2016-11-08T09:49:57Z 9506 bytes transferred. (165 KB/s) (56 ms)
2016-11-08T09:49:57Z Getting properties (Date modified, Date created) of "::{82AA9188-44E0-40B9-B956-43A10C315B4F}\::{6DDED4A4-3958-49C1-8E87-D9CE657B977A}/LOGS/20160921 231303 01 {298B0B39-D344-45DC-87F3-F33382EEE806}.log"
2016-11-08T09:49:57Z Item: Date modified=2016-09-21T21:14:00
2016-11-08T09:49:57Z Setting properties (Date modified) of "E:\LOGS\20160921 231303 01 {298B0B39-D344-45DC-87F3-F33382EEE806}.log"
2016-11-08T09:49:57Z Getting properties (Size) of "::{82AA9188-44E0-40B9-B956-43A10C315B4F}\::{6DDED4A4-3958-49C1-8E87-D9CE657B977A}/LOGS/20160921 231303 01 {298B0B39-D344-45DC-87F3-F33382EEE806}.log"
2016-11-08T09:49:57Z Item: Size=9506
2016-11-08T09:49:57Z Using cached properties
2016-11-08T09:49:57Z Getting properties (Size) of "E:\LOGS\20160921 231303 01 {298B0B39-D344-45DC-87F3-F33382EEE806}.log"
2016-11-08T09:49:57Z Item: Size=9506
2016-11-08T09:49:57Z Source File Size=9506, Destination file size=9506
2016-11-08T09:49:57Z Operation end
 
 
Thanks a lot

[BruceLee]

Another thing I'm experiencing is a timeshift of the remote files wiht one hour ahead. It seems to have to do with DST.
Tested both (Automatci and correct timezone) settings in favorite. No change...thanks.
Just have seen this after a smartftp restart. this has been running for days and was started before time changes due to DST.

[mb]

We are unable to reproduce the issue you have reported. We have tested it with the .txt file on our ftp server (IIS): ftp.smartftp.com
 
The log looks like this in our case:
 
<blockquote class="ipsBlockquote">
2016-11-08T11:58:18Z Item: Size=109, Date modified=2012-08-12T03:25:55
2016-11-08T11:58:18Z Source file: Size=109, Date modified=2012-08-12T03:25:55
2016-11-08T11:58:18Z Destination file: Size=109, Date modified=2012-08-12T03:25:55
2016-11-08T11:58:18Z Rule "IF Destination Time=Equal AND Size=Equal AND Transfer=No matter THEN Skip" matched. Action="Skip".
2016-11-08T11:58:18Z Skipping file "/This is a DEMO server.txt". Reason: File exist rules.
 
</blockquote>
 
 
Did you change the file exist rules?

[BruceLee]

thanks. no the rules are the default ones.
It is really strange. I'm deleting some files now and will resync.
How to deal with DST?

[mb]

>sync
You can also try it with the same .txt file on our server (ftp.smartftp.com)
 
>dst
Do you see the time discrepancy in the remote browser (file view)? One problem is that the IIS FTP server still does not support MLSD and therefore returns the times in the directory listing in the local system time. This time is then converted to UTC. For the keep file time/get file time function, MDTM is used which returns the time in UTC and you should not see any issues because in this case the time is never converted.

[BruceLee]

thanks. I have set it back to Automatic timezone. I have seen the discrepancy in thr remote browser. somehow it is now showing the files correctly. but it still overwrites. the directory has 10K+ of files. maybe thats the problem. I have excluded this directory completely to go on with testing.
the other dirs work well. its really strange. for testing i have the cache en- or disabled...doesnt change anything.
 
Another question: If during syncing a file that was in queue to be synced is deleted in source and therefore cannot and also should not be synced anymore...how to cover this case the item NOT to stuck in queue with "the system cannot find the file specified"?
 
Sidenote: importing a schedule over smartftp (with or without admin privileges) on WinServer2012R2x64 lets smartftp crash.
Tested by creating a schedule the intended way over smartftp and then export it, import it.

Thanks

[mb]

>overwrite
Test it first with the .txt file on our server: ftp.smartftp.com
Then create a new connection (new favorite) and test it with a single file on your server.
 
>stuck in queue with "the system cannot find the file specified"?
You need to rethink your process if you have concurrent writers/readers.
 
>schedule import
Fixed. Bug fix will be in the next build.

[BruceLee]

thanks...but like I wrote it is working for certain directories on our server. so it will work with your txt file and your server too. anyway.
 
regarding your server. I dont want to open the huge passive port range from LAN to public you might have setup in your IIS.
So I have downloaded your file and tested.. it works. Single file with new favorite and old favorite works. Like I already wrote. But it doesnt for this "special" directory. Which is only special by the meaning of amount of files (10K+) with ~16KB each in average.
Name pattern is 20160923 160000 01 {DC07C408-378E-410E-8062-1709E705FBE6}.log
 
Have you maybe tested with 10K+ files?
Anything regarding my questions? Thanks
 

[mb]

>overwrite
the file name and the number of files is irrelevant. Does it happen for all files in this "special folder" or randomly for some files (or is it always the same file)?
 
Your other questions have already been addressed in the last reply.

[BruceLee]

thanks.
 
>overwrite
the file name and the number of files is irrelevant. Does it happen for all files in this "special folder" or randomly for some files (or is it always the same file)?
All files. not random.
 
>stuck in queue with "the system cannot find the file specified"?
You need to rethink your process if you have concurrent writers/readers.
thanks. its just the case if the sync of previous files take so long that another process has deleted some old files that do not have to be synced anymore.
is there an option to kick an item out if queue after X failed retries?