Problems understanding rules

I've been trying to find a way to upload ONLY changed files to my website(s).

But after reading the posts and how-tos, I still can't figure out how to get SFTP to ONLY upload files that DON'T already exist OR which are different.

Each time I try to upload using specific rules,
- EITHER ALL files are overwritten regardless of their date and timestamp,
- OR NO modified files are uploaded, only files that don't already exist!

I've just tried for nearly 2 hours trying to figure out how to configure the settings so that they do this incredibly simple and useful function, but I'm getting nowhere. I did figure out why uploads based on the timestamp don't work : the server uses it's own (incorrectly set) timezone to set the file date and time (this happens with 3 separate servers, none of which I am able to change since they're hosted by my various customers' ISPs). So I can't use SFTP's date/time checking options, they are effectively useless. Maybe that would be an option to add to a later version - if I can tell SFTP to calculate destination file age based on the destination server's timezone and clock, that would be a pretty cool way to avoid having to manually copy files. But I digress...

I found a mention in one of the settings options that seems to offer a way to determine if a remote file is intact (in the integrity option page), but it seems to imply that it's only used to verify the target file is the target file, and isn't used to actually COMPARE file contents, just EVALUATE file contents.

So I would really appreciate any assistance or pointers to show me how, exactly, I can set up a favourite site so that no matter what files I upload, ONLY files that are DIFFERENT OR files that DON'T ALREADY EXIST on the server are overwritten?

If I'm missing something blindingly obvious, please be gentle with me, it's been a long day.

Please post the log from the transfer. To get it double click the file in the transfer queue.

By default SmartFTP does only upload the modified files.

Regards,
Mat

To get the log from the transfer queue:
1. Stop the transfer queue
2. Add the files to the transfer queue
3. Double click the first file
4. The log window opens
5. Start the Transfer Queue
6. Copy the log from the log window

Thanks for your persistence with this!

By way of explanation for log panes, my current workspace consists of a left pane (my local site directory structure) and a right pane (the remote site directory structure). Both structures have sync nav on. In the right (site) pane, I also have the log view open, so I see all the NOPs and so on.

I reset the default file exists rule to "Ask" to try and help me figure out how SFTP is identifying the duplicate files, but that's just confused me more, as the overwrite prompt only shows the date/time have changed (they haven't, the server's date/time and timezone setting is just different to mine), there's no indication that the content is identical, and the checkbox value is ignored (i.e. the "Ask" setting in the site's preferences isn't reset by the checkbox). So I'll skip that for now.

Using the "Default Rules" setting, I get *exactly* the same prompts and commands as using "Ask" and selecting "Overwrite". I haven't changed any default rules that I'm aware of, the only things I've tried to change are the favourite's rules in the favourite's property settings, and I only modify those when I'm not connected to the site.

I can see there seems to be a problem in the transfer queue log, with the MDTM and MFMT commands, but I have no idea why the MDTM works in the main log command stream, but not in the tq log stream.

First, I'll transfer the exact same file I just transferred less than a minute ago.
In the right pane log view, when I select the duplicate file for transfer to the remote site, I see:

The transfer queue log is:


And then in the site log pane, I see:

(N.B. I DON'T see the failed commands from the transfer log in the site log and vice-versa, so I'm not just cutting and pasting the bits I think are important, the only data I've removed is the login and IP/user details, and the IPs are modified to protect the innocent
Now, transferring a different file, but with exactly the same size and local date/time stamp as the previous file, in the site log pane I see:
Again, there is no indication of the content being changed, only the same type of date/time stamp mismatch as with the original file I transferred previously.

Anyway, in the transfer queue log, I now get:
And finally in the site log :

I'm sorry to keep saying this, but the time/date and size checks are completely irrelevant to me. I ONLY want to overwrite files with the same name that have different content.

Using the hash option for identical files in the favourite's identical file settings page, doesn't appear to do anything - there are no extra commands visible in either log pane when that option is checked.

So I still can't seem to find a way to use any of the rules in a meaningful way. If I choose "no matter" for size and time, I only get the option to overwrite or skip, and SFTP doesn't seem to take the file content into consideration, so the rule matching bypasses the content check.

OTOH, if I use "unknown" for the size/time checks, the rule always fails, as the time and size CAN be determined, and the remote file is overwritten or skipped - again, regardless of the content!

If I use any other combination of size and/or time rules, again, the content check is skipped, and ONLY the SIZE or the TIME are used to determine the action.

So the SFTP overwrite/skip logic always seems to bypass the file content, which is the only variable I want to base the uploads on. Or am I missing something much more obvious and simple?

I don't mind the performance penalty involved with determining file contents with identical sizes with the hash algorithm - I don't want a FAST site update, I want a CORRECT site update! I hope that makes sense.

Since I manage a few different client sites with different ISPs, all the servers are slightly different, and all have different date/time settings, so I CAN'T just use a single time/date offset, as that won't work across daylight savings changes, and I can't use a per-site offset, as that is invalidated by server rebuilds and daylight savings and so on. Since I have no control over the remote servers, the time and date stamps are completely irrelevant and useless. That's why I want to use the content/size as the criteria for overwriting.

If you would like me to set up and try some site rules to see if we can get SFTP working the way I need, please let me know and I'll get back to you asap.

I should also note that when renaming a file in my local files pane using SFTP, SFTP displays the modified date and time, but the "modified" timestamp for the file on the NTFS volume doesn't change. But that's another issue for later I guess! One step at a time...

I'm also happy to continue trying to find out why the default rules don't seem to work the way I expect, if it helps sort someone else out too that's no problem.

But my immediate focus is getting SFTP working with my client sites, as I'm currently doing bulk uploads for ALL the sites!

Cheers,
PCPete

Hello Pete ..

The main problem is that your FTP server neither offers a feature to set the date/time of a remote file nor a feature to calculate the hash value of a file for the integrity check. Why is the date/time important? Since many servers do not support an integrity function SmartFTP "tags" the remote file with the same date/time as the source file after the transfer completes. This way if it sees the same file again and both last modified time values match it will skip the file by default. There is no guarantee that both files (content) are equal but in this particular case it is very safe to assume that they are.

Conclusion: SmartFTP does the best it can considering the features the server offers. The best you can do is to replace the FTP server software with a modern FTP server. E.g on Unix we can recommend the latest version of proftpd with the optional file integrity check module: https://www.smartftp.com/oss/proftpd/

Regards,
Mat