overwrite in loop
-
- Posts: 23
- Joined: Fri Feb 14, 2014 9:42 am
overwrite in loop
Hi,
I'm currently testing the enterprise edition.
I want to synchronize files from a FTP server down to a local drive.
Since it's a one way sync with delete it needs to check if the file is already existing locally.
that works well....but
it keeps resyncing the files again and again. Minutes, hours and datestamp are correct.
Simple ftp transfer in the internal network.
The favorite settings are set to "enable" in "keep modify time" settings.
Remote FTP server is an IIS 8.5 with the following supported commands:
ABOR
ACCT
ADAT *
ALLO
APPE
AUTH
CCC
CDUP
CWD
DELE
ENC *
EPRT
EPSV
FEAT
HELP
HOST
LANG
LIST
MDTM
MIC *
MKD
MODE
NLST
NOOP
OPTS
PASS
PASV
PBSZ
PORT
PROT
PWD
QUIT
REIN
REST
RETR
RMD
RNFR
RNTO
SITE
SIZE
SMNT
STAT
STOR
STOU
STRU
SYST
TYPE
USER
XCUP
XCWD
XMKD
XPWD
XRMD
This is the first sync before the file exists locally.
2016-11-08T09:49:51Z Operation begin
2016-11-08T09:49:51Z Copying file. Source="::{82AA9188-44E0-40B9-B956-43A10C315B4F}\::{6DDED4A4-3958-49C1-8E87-D9CE657B977A}/LOGS/20160921 231303 01 {298B0B39-D344-45DC-87F3-F33382EEE806}.log", Destination="E:\LOGS\20160921 231303 01 {298B0B39-D344-45DC-87F3-F33382EEE806}.log"
2016-11-08T09:49:51Z Getting properties (Size, Date modified) of "::{82AA9188-44E0-40B9-B956-43A10C315B4F}\::{6DDED4A4-3958-49C1-8E87-D9CE657B977A}/LOGS/20160921 231303 01 {298B0B39-D344-45DC-87F3-F33382EEE806}.log"
2016-11-08T09:49:51Z Item: Size=9506, Date modified=2016-09-21T21:14:00
2016-11-08T09:49:51Z Transfer restarting at position 0.
2016-11-08T09:49:51Z 9506 bytes transferred. (97,7 KB/s) (95 ms)
2016-11-08T09:49:51Z Getting properties (Date modified, Date created) of "::{82AA9188-44E0-40B9-B956-43A10C315B4F}\::{6DDED4A4-3958-49C1-8E87-D9CE657B977A}/LOGS/20160921 231303 01 {298B0B39-D344-45DC-87F3-F33382EEE806}.log"
2016-11-08T09:49:51Z Item: Date modified=2016-09-21T21:14:00
2016-11-08T09:49:51Z Setting properties (Date modified) of "E:\LOGS\20160921 231303 01 {298B0B39-D344-45DC-87F3-F33382EEE806}.log"
2016-11-08T09:49:51Z Getting properties (Size) of "::{82AA9188-44E0-40B9-B956-43A10C315B4F}\::{6DDED4A4-3958-49C1-8E87-D9CE657B977A}/LOGS/20160921 231303 01 {298B0B39-D344-45DC-87F3-F33382EEE806}.log"
2016-11-08T09:49:51Z Item: Size=9506
2016-11-08T09:49:51Z Using cached properties
2016-11-08T09:49:51Z Getting properties (Size) of "E:\LOGS\20160921 231303 01 {298B0B39-D344-45DC-87F3-F33382EEE806}.log"
2016-11-08T09:49:51Z Item: Size=9506
2016-11-08T09:49:51Z Source File Size=9506, Destination file size=9506
2016-11-08T09:49:51Z Operation end
And this is the second that should skip the file (default ruleset of smartftp is in use with checked setting of hash values too):
2016-11-08T09:49:57Z Operation begin
2016-11-08T09:49:57Z Copying file. Source="::{82AA9188-44E0-40B9-B956-43A10C315B4F}\::{6DDED4A4-3958-49C1-8E87-D9CE657B977A}/LOGS/20160921 231303 01 {298B0B39-D344-45DC-87F3-F33382EEE806}.log", Destination="E:\LOGS\20160921 231303 01 {298B0B39-D344-45DC-87F3-F33382EEE806}.log"
2016-11-08T09:49:57Z Getting properties (Size, Date modified) of "::{82AA9188-44E0-40B9-B956-43A10C315B4F}\::{6DDED4A4-3958-49C1-8E87-D9CE657B977A}/LOGS/20160921 231303 01 {298B0B39-D344-45DC-87F3-F33382EEE806}.log"
2016-11-08T09:49:57Z Item: Size=9506, Date modified=2016-09-21T21:14:00
2016-11-08T09:49:57Z Getting properties (Size, Date modified) of "E:\LOGS\20160921 231303 01 {298B0B39-D344-45DC-87F3-F33382EEE806}.log"
2016-11-08T09:49:57Z Item: Size=9506, Date modified=2016-09-21T21:14:00
2016-11-08T09:49:57Z Source file: Size=9506, Date modified=2016-09-21T21:14:00
2016-11-08T09:49:57Z Destination file: Size=9506, Date modified=2016-09-21T21:14:00
2016-11-08T09:49:57Z No rule matched. Default action="Overwrite".
2016-11-08T09:49:57Z Transfer restarting at position 0.
2016-11-08T09:49:57Z 9506 bytes transferred. (165 KB/s) (56 ms)
2016-11-08T09:49:57Z Getting properties (Date modified, Date created) of "::{82AA9188-44E0-40B9-B956-43A10C315B4F}\::{6DDED4A4-3958-49C1-8E87-D9CE657B977A}/LOGS/20160921 231303 01 {298B0B39-D344-45DC-87F3-F33382EEE806}.log"
2016-11-08T09:49:57Z Item: Date modified=2016-09-21T21:14:00
2016-11-08T09:49:57Z Setting properties (Date modified) of "E:\LOGS\20160921 231303 01 {298B0B39-D344-45DC-87F3-F33382EEE806}.log"
2016-11-08T09:49:57Z Getting properties (Size) of "::{82AA9188-44E0-40B9-B956-43A10C315B4F}\::{6DDED4A4-3958-49C1-8E87-D9CE657B977A}/LOGS/20160921 231303 01 {298B0B39-D344-45DC-87F3-F33382EEE806}.log"
2016-11-08T09:49:57Z Item: Size=9506
2016-11-08T09:49:57Z Using cached properties
2016-11-08T09:49:57Z Getting properties (Size) of "E:\LOGS\20160921 231303 01 {298B0B39-D344-45DC-87F3-F33382EEE806}.log"
2016-11-08T09:49:57Z Item: Size=9506
2016-11-08T09:49:57Z Source File Size=9506, Destination file size=9506
2016-11-08T09:49:57Z Operation end
Thanks a lot
I'm currently testing the enterprise edition.
I want to synchronize files from a FTP server down to a local drive.
Since it's a one way sync with delete it needs to check if the file is already existing locally.
that works well....but
it keeps resyncing the files again and again. Minutes, hours and datestamp are correct.
Simple ftp transfer in the internal network.
The favorite settings are set to "enable" in "keep modify time" settings.
Remote FTP server is an IIS 8.5 with the following supported commands:
ABOR
ACCT
ADAT *
ALLO
APPE
AUTH
CCC
CDUP
CWD
DELE
ENC *
EPRT
EPSV
FEAT
HELP
HOST
LANG
LIST
MDTM
MIC *
MKD
MODE
NLST
NOOP
OPTS
PASS
PASV
PBSZ
PORT
PROT
PWD
QUIT
REIN
REST
RETR
RMD
RNFR
RNTO
SITE
SIZE
SMNT
STAT
STOR
STOU
STRU
SYST
TYPE
USER
XCUP
XCWD
XMKD
XPWD
XRMD
This is the first sync before the file exists locally.
2016-11-08T09:49:51Z Operation begin
2016-11-08T09:49:51Z Copying file. Source="::{82AA9188-44E0-40B9-B956-43A10C315B4F}\::{6DDED4A4-3958-49C1-8E87-D9CE657B977A}/LOGS/20160921 231303 01 {298B0B39-D344-45DC-87F3-F33382EEE806}.log", Destination="E:\LOGS\20160921 231303 01 {298B0B39-D344-45DC-87F3-F33382EEE806}.log"
2016-11-08T09:49:51Z Getting properties (Size, Date modified) of "::{82AA9188-44E0-40B9-B956-43A10C315B4F}\::{6DDED4A4-3958-49C1-8E87-D9CE657B977A}/LOGS/20160921 231303 01 {298B0B39-D344-45DC-87F3-F33382EEE806}.log"
2016-11-08T09:49:51Z Item: Size=9506, Date modified=2016-09-21T21:14:00
2016-11-08T09:49:51Z Transfer restarting at position 0.
2016-11-08T09:49:51Z 9506 bytes transferred. (97,7 KB/s) (95 ms)
2016-11-08T09:49:51Z Getting properties (Date modified, Date created) of "::{82AA9188-44E0-40B9-B956-43A10C315B4F}\::{6DDED4A4-3958-49C1-8E87-D9CE657B977A}/LOGS/20160921 231303 01 {298B0B39-D344-45DC-87F3-F33382EEE806}.log"
2016-11-08T09:49:51Z Item: Date modified=2016-09-21T21:14:00
2016-11-08T09:49:51Z Setting properties (Date modified) of "E:\LOGS\20160921 231303 01 {298B0B39-D344-45DC-87F3-F33382EEE806}.log"
2016-11-08T09:49:51Z Getting properties (Size) of "::{82AA9188-44E0-40B9-B956-43A10C315B4F}\::{6DDED4A4-3958-49C1-8E87-D9CE657B977A}/LOGS/20160921 231303 01 {298B0B39-D344-45DC-87F3-F33382EEE806}.log"
2016-11-08T09:49:51Z Item: Size=9506
2016-11-08T09:49:51Z Using cached properties
2016-11-08T09:49:51Z Getting properties (Size) of "E:\LOGS\20160921 231303 01 {298B0B39-D344-45DC-87F3-F33382EEE806}.log"
2016-11-08T09:49:51Z Item: Size=9506
2016-11-08T09:49:51Z Source File Size=9506, Destination file size=9506
2016-11-08T09:49:51Z Operation end
And this is the second that should skip the file (default ruleset of smartftp is in use with checked setting of hash values too):
2016-11-08T09:49:57Z Operation begin
2016-11-08T09:49:57Z Copying file. Source="::{82AA9188-44E0-40B9-B956-43A10C315B4F}\::{6DDED4A4-3958-49C1-8E87-D9CE657B977A}/LOGS/20160921 231303 01 {298B0B39-D344-45DC-87F3-F33382EEE806}.log", Destination="E:\LOGS\20160921 231303 01 {298B0B39-D344-45DC-87F3-F33382EEE806}.log"
2016-11-08T09:49:57Z Getting properties (Size, Date modified) of "::{82AA9188-44E0-40B9-B956-43A10C315B4F}\::{6DDED4A4-3958-49C1-8E87-D9CE657B977A}/LOGS/20160921 231303 01 {298B0B39-D344-45DC-87F3-F33382EEE806}.log"
2016-11-08T09:49:57Z Item: Size=9506, Date modified=2016-09-21T21:14:00
2016-11-08T09:49:57Z Getting properties (Size, Date modified) of "E:\LOGS\20160921 231303 01 {298B0B39-D344-45DC-87F3-F33382EEE806}.log"
2016-11-08T09:49:57Z Item: Size=9506, Date modified=2016-09-21T21:14:00
2016-11-08T09:49:57Z Source file: Size=9506, Date modified=2016-09-21T21:14:00
2016-11-08T09:49:57Z Destination file: Size=9506, Date modified=2016-09-21T21:14:00
2016-11-08T09:49:57Z No rule matched. Default action="Overwrite".
2016-11-08T09:49:57Z Transfer restarting at position 0.
2016-11-08T09:49:57Z 9506 bytes transferred. (165 KB/s) (56 ms)
2016-11-08T09:49:57Z Getting properties (Date modified, Date created) of "::{82AA9188-44E0-40B9-B956-43A10C315B4F}\::{6DDED4A4-3958-49C1-8E87-D9CE657B977A}/LOGS/20160921 231303 01 {298B0B39-D344-45DC-87F3-F33382EEE806}.log"
2016-11-08T09:49:57Z Item: Date modified=2016-09-21T21:14:00
2016-11-08T09:49:57Z Setting properties (Date modified) of "E:\LOGS\20160921 231303 01 {298B0B39-D344-45DC-87F3-F33382EEE806}.log"
2016-11-08T09:49:57Z Getting properties (Size) of "::{82AA9188-44E0-40B9-B956-43A10C315B4F}\::{6DDED4A4-3958-49C1-8E87-D9CE657B977A}/LOGS/20160921 231303 01 {298B0B39-D344-45DC-87F3-F33382EEE806}.log"
2016-11-08T09:49:57Z Item: Size=9506
2016-11-08T09:49:57Z Using cached properties
2016-11-08T09:49:57Z Getting properties (Size) of "E:\LOGS\20160921 231303 01 {298B0B39-D344-45DC-87F3-F33382EEE806}.log"
2016-11-08T09:49:57Z Item: Size=9506
2016-11-08T09:49:57Z Source File Size=9506, Destination file size=9506
2016-11-08T09:49:57Z Operation end
Thanks a lot
-
- Posts: 23
- Joined: Fri Feb 14, 2014 9:42 am
Re: overwrite in loop
Another thing I'm experiencing is a timeshift of the remote files wiht one hour ahead. It seems to have to do with DST.
Tested both (Automatci and correct timezone) settings in favorite. No change...thanks.
Just have seen this after a smartftp restart. this has been running for days and was started before time changes due to DST.
Tested both (Automatci and correct timezone) settings in favorite. No change...thanks.
Just have seen this after a smartftp restart. this has been running for days and was started before time changes due to DST.
Last edited by BruceLee on Tue Nov 08, 2016 11:11 am, edited 1 time in total.
-
- Posts: 12031
- Joined: Thu Oct 11, 2001 1:24 pm
- Display name: Mat (SmartFTP)
- Contact:
Re: overwrite in loop
We are unable to reproduce the issue you have reported. We have tested it with the .txt file on our ftp server (IIS): ftp.smartftp.com
The log looks like this in our case:
<blockquote class="ipsBlockquote">
2016-11-08T11:58:18Z Item: Size=109, Date modified=2012-08-12T03:25:55
2016-11-08T11:58:18Z Source file: Size=109, Date modified=2012-08-12T03:25:55
2016-11-08T11:58:18Z Destination file: Size=109, Date modified=2012-08-12T03:25:55
2016-11-08T11:58:18Z Rule "IF Destination Time=Equal AND Size=Equal AND Transfer=No matter THEN Skip" matched. Action="Skip".
2016-11-08T11:58:18Z Skipping file "/This is a DEMO server.txt". Reason: File exist rules.
</blockquote>
Did you change the file exist rules?
The log looks like this in our case:
<blockquote class="ipsBlockquote">
2016-11-08T11:58:18Z Item: Size=109, Date modified=2012-08-12T03:25:55
2016-11-08T11:58:18Z Source file: Size=109, Date modified=2012-08-12T03:25:55
2016-11-08T11:58:18Z Destination file: Size=109, Date modified=2012-08-12T03:25:55
2016-11-08T11:58:18Z Rule "IF Destination Time=Equal AND Size=Equal AND Transfer=No matter THEN Skip" matched. Action="Skip".
2016-11-08T11:58:18Z Skipping file "/This is a DEMO server.txt". Reason: File exist rules.
</blockquote>
Did you change the file exist rules?
Last edited by mb on Tue Nov 08, 2016 12:17 pm, edited 1 time in total.
-
- Posts: 23
- Joined: Fri Feb 14, 2014 9:42 am
Re: overwrite in loop
thanks. no the rules are the default ones.
It is really strange. I'm deleting some files now and will resync.
How to deal with DST?
It is really strange. I'm deleting some files now and will resync.
How to deal with DST?
-
- Posts: 12031
- Joined: Thu Oct 11, 2001 1:24 pm
- Display name: Mat (SmartFTP)
- Contact:
Re: overwrite in loop
>sync
You can also try it with the same .txt file on our server (ftp.smartftp.com)
>dst
Do you see the time discrepancy in the remote browser (file view)? One problem is that the IIS FTP server still does not support MLSD and therefore returns the times in the directory listing in the local system time. This time is then converted to UTC. For the keep file time/get file time function, MDTM is used which returns the time in UTC and you should not see any issues because in this case the time is never converted.
You can also try it with the same .txt file on our server (ftp.smartftp.com)
>dst
Do you see the time discrepancy in the remote browser (file view)? One problem is that the IIS FTP server still does not support MLSD and therefore returns the times in the directory listing in the local system time. This time is then converted to UTC. For the keep file time/get file time function, MDTM is used which returns the time in UTC and you should not see any issues because in this case the time is never converted.
-
- Posts: 23
- Joined: Fri Feb 14, 2014 9:42 am
Re: overwrite in loop
thanks. I have set it back to Automatic timezone. I have seen the discrepancy in thr remote browser. somehow it is now showing the files correctly. but it still overwrites. the directory has 10K+ of files. maybe thats the problem. I have excluded this directory completely to go on with testing.
the other dirs work well. its really strange. for testing i have the cache en- or disabled...doesnt change anything.
Another question: If during syncing a file that was in queue to be synced is deleted in source and therefore cannot and also should not be synced anymore...how to cover this case the item NOT to stuck in queue with "the system cannot find the file specified"?
Sidenote: importing a schedule over smartftp (with or without admin privileges) on WinServer2012R2x64 lets smartftp crash.
Tested by creating a schedule the intended way over smartftp and then export it, import it.
Thanks
the other dirs work well. its really strange. for testing i have the cache en- or disabled...doesnt change anything.
Another question: If during syncing a file that was in queue to be synced is deleted in source and therefore cannot and also should not be synced anymore...how to cover this case the item NOT to stuck in queue with "the system cannot find the file specified"?
Sidenote: importing a schedule over smartftp (with or without admin privileges) on WinServer2012R2x64 lets smartftp crash.
Tested by creating a schedule the intended way over smartftp and then export it, import it.
Thanks
Last edited by BruceLee on Tue Nov 08, 2016 3:30 pm, edited 1 time in total.
-
- Posts: 12031
- Joined: Thu Oct 11, 2001 1:24 pm
- Display name: Mat (SmartFTP)
- Contact:
Re: overwrite in loop
>overwrite
Test it first with the .txt file on our server: ftp.smartftp.com
Then create a new connection (new favorite) and test it with a single file on your server.
>stuck in queue with "the system cannot find the file specified"?
You need to rethink your process if you have concurrent writers/readers.
>schedule import
Fixed. Bug fix will be in the next build.
Test it first with the .txt file on our server: ftp.smartftp.com
Then create a new connection (new favorite) and test it with a single file on your server.
>stuck in queue with "the system cannot find the file specified"?
You need to rethink your process if you have concurrent writers/readers.
>schedule import
Fixed. Bug fix will be in the next build.
Last edited by mb on Tue Nov 08, 2016 3:56 pm, edited 1 time in total.
-
- Posts: 23
- Joined: Fri Feb 14, 2014 9:42 am
Re: overwrite in loop
thanks...but like I wrote it is working for certain directories on our server. so it will work with your txt file and your server too. anyway.
regarding your server. I dont want to open the huge passive port range from LAN to public you might have setup in your IIS.
So I have downloaded your file and tested.. it works. Single file with new favorite and old favorite works. Like I already wrote. But it doesnt for this "special" directory. Which is only special by the meaning of amount of files (10K+) with ~16KB each in average.
Name pattern is 20160923 160000 01 {DC07C408-378E-410E-8062-1709E705FBE6}.log
Have you maybe tested with 10K+ files?
Anything regarding my questions? Thanks
regarding your server. I dont want to open the huge passive port range from LAN to public you might have setup in your IIS.
So I have downloaded your file and tested.. it works. Single file with new favorite and old favorite works. Like I already wrote. But it doesnt for this "special" directory. Which is only special by the meaning of amount of files (10K+) with ~16KB each in average.
Name pattern is 20160923 160000 01 {DC07C408-378E-410E-8062-1709E705FBE6}.log
Have you maybe tested with 10K+ files?
Anything regarding my questions? Thanks
Last edited by BruceLee on Tue Nov 08, 2016 4:29 pm, edited 1 time in total.
-
- Posts: 12031
- Joined: Thu Oct 11, 2001 1:24 pm
- Display name: Mat (SmartFTP)
- Contact:
Re: overwrite in loop
>overwrite
the file name and the number of files is irrelevant. Does it happen for all files in this "special folder" or randomly for some files (or is it always the same file)?
Your other questions have already been addressed in the last reply.
the file name and the number of files is irrelevant. Does it happen for all files in this "special folder" or randomly for some files (or is it always the same file)?
Your other questions have already been addressed in the last reply.
-
- Posts: 23
- Joined: Fri Feb 14, 2014 9:42 am
Re: overwrite in loop
thanks.
>overwrite
the file name and the number of files is irrelevant. Does it happen for all files in this "special folder" or randomly for some files (or is it always the same file)?
All files. not random.
>stuck in queue with "the system cannot find the file specified"?
You need to rethink your process if you have concurrent writers/readers.
thanks. its just the case if the sync of previous files take so long that another process has deleted some old files that do not have to be synced anymore.
is there an option to kick an item out if queue after X failed retries?
>overwrite
the file name and the number of files is irrelevant. Does it happen for all files in this "special folder" or randomly for some files (or is it always the same file)?
All files. not random.
>stuck in queue with "the system cannot find the file specified"?
You need to rethink your process if you have concurrent writers/readers.
thanks. its just the case if the sync of previous files take so long that another process has deleted some old files that do not have to be synced anymore.
is there an option to kick an item out if queue after X failed retries?