Jump to content


Photo

Improve folder monitoring by implementing delay


  • Please log in to reply
19 replies to this topic

#1 CraigBee

CraigBee
  • Members
  • 19 posts

Posted 11 November 2010 - 04:58 PM

When a large file (~2GB) is copied to a folder being monitored by SmartFTP, it may upload several times. I have SmartFTP configured to use 5 upload threads. When a 2GB file is copied to the NAS folder being monitored, SmartFTP will detect the file and start an upload before the file copy is finished. The file then 'changes' as more of the file is copied, and SmartFTP will start a second upload.

I think this might be avoidable by waiting the monitored folder to stop changing before processing the monitored folder. The logic could be something like this:
Step 1: Wait for change in monitored folder
Step 2: Remember current state of monitored folder
Step 3: Wait 20 seconds
Step 4: Check for change in monitored folder since step 2, if changed, goto Step 2
Step 5: Process monitored folder according to rules

Something like this would allow SmartFTP to wait for large files to finish being copied to the monitored folder before starting the upload. It would then only upload 1 time instead of multiple times.

Craig

#2 mb

mb

    Developer

  • Administrators
  • 11527 posts
  • Gender:
    Male
  • Location:
    Worldwide

Posted 11 November 2010 - 05:02 PM

Duplicate transfer queue items should automatically be ignored. I will do some investigations why this is apparently not the case.

#3 CraigBee

CraigBee
  • Members
  • 19 posts

Posted 11 November 2010 - 07:30 PM

I'm on version 4.0.1135.0 (64-bit). I'm also forcing all files to lower case. A bug related to file matching with forced lower case was fixed in 4.0.1134.0. Perhaps it is related.

Edited by CraigBee, 11 November 2010 - 07:31 PM.


#4 mb

mb

    Developer

  • Administrators
  • 11527 posts
  • Gender:
    Male
  • Location:
    Worldwide

Posted 11 November 2010 - 07:40 PM

Please try it again with the latest version: http://www.smartftp.com/download

#5 CraigBee

CraigBee
  • Members
  • 19 posts

Posted 08 December 2010 - 08:29 PM

Please try it again with the latest version: http://www.smartftp.com/download


When I check the virtual machine every few days, I often find it is crashed and a window pops up says "SmartFTP Client has stopped working. Windows can check online for a solution to the problem and try to restart the program.". I did not get this before updating to the most recent version. What log files or data can I provide to help analyze this crash?

The duplicate transfers are gone though :o

Craig

Edited by CraigBee, 08 December 2010 - 08:53 PM.


#6 mb

mb

    Developer

  • Administrators
  • 11527 posts
  • Gender:
    Male
  • Location:
    Worldwide

Posted 08 December 2010 - 09:43 PM

Please install the latest version and try again:
32-bit
http://www.smartftp....et/SFTPBeta.exe

64-bit
http://www.smartftp..../SFTPBeta64.exe

If the problem persits and you have a way to reproduce it we can give you further instructions. Thanks.

#7 CraigBee

CraigBee
  • Members
  • 19 posts

Posted 14 December 2010 - 02:29 PM

It still crashes. I have isolated it to some extent. The crash only occurs when connecting to a local Windows IIS FTP server. It seems random. Over long enough period, sometimes seconds, sometimes hours, it will eventually crash when uploading to the Windows IIS FTP server. The crash has never occurred when transferring to other FTP servers that are not local and of unknown software.

#8 mb

mb

    Developer

  • Administrators
  • 11527 posts
  • Gender:
    Male
  • Location:
    Worldwide

Posted 14 December 2010 - 07:18 PM

If you can reproduce it in a relativ short time (minutes) please create a crash dump as described in this article: http://www.smartftp....orts-f2594.html

#9 CraigBee

CraigBee
  • Members
  • 19 posts

Posted 15 December 2010 - 07:54 PM

If you can reproduce it in a relativ short time (minutes) please create a crash dump as described in this article: http://www.smartftp....orts-f2594.html


I was able to reproduce a crash twice within a few minutes of starting SmartFTP and emailed the 10K & 16K C:\Dump directory to support@smartftp.com. If that email can't receive attachments, let me know and I will find a place to upload the file.

After more testing, I am able to reproduce a crash on demand with the following steps using SmartFTP build 1148:
1. Start SmartFTP, ensure the queue and schedule is clear.
2. Add new item to queue to copy a folder with delete from NAS to any FTP server
3. Right click and hit schedule
4. Add Monitor Folder - instant crash

Craig

Edited by CraigBee, 15 December 2010 - 08:30 PM.


#10 mb

mb

    Developer

  • Administrators
  • 11527 posts
  • Gender:
    Male
  • Location:
    Worldwide

Posted 15 December 2010 - 10:24 PM

Thanks. Can you post your system information from the menu: Help->About "System Information"?

I couldn't reproduce it right away. What folder are you monitoring? UNC or "local" folder?

#11 CraigBee

CraigBee
  • Members
  • 19 posts

Posted 16 December 2010 - 05:18 PM

Uhhhg. I can get SmartFTP to crash within 5 seconds... when NOT running the debug software. When running debug, it won't crash. (If a computer crashes and no debug is there to record it, did it actually crash?). Perhaps it is a timing issue, because SmartFTP runs much much slower with the debug software logging. Also, it seems more likely to crash in the non-administrator account. Here is the info from the non-admin account where I can reproduce the crash (when not running debug. No really, it crashes!):
+- System -----------------------------
Microsoft Windows 2008 Server Standard Edition (full installation)
Service Pack 2 (Build 6002)

CPU Speed : 2535 MHz
Total Memory : 8190 MB
Free Memory : 7270 MB

+- SmartFTP ---------------------------
Version : 4.0.1148.0
Time Stamp : 2010-12-07 22:26:38
Platform : x64
Id : 400128942
Maintenance : 2011-08-12
Days in use : 119

+- Language ---------------------------
en-US

+- Internet Explorer ------------------
Version : 8.0.6001.18943

+- Winsock ----------------------------
Winsock : 2.2

#12 mb

mb

    Developer

  • Administrators
  • 11527 posts
  • Gender:
    Male
  • Location:
    Worldwide

Posted 16 December 2010 - 06:07 PM

And how exactly do you make it crash? Is adding a new monitored folder enough?
We can only fix it with a crash dump or if we can actually reproduce it here.

#13 CraigBee

CraigBee
  • Members
  • 19 posts

Posted 16 December 2010 - 06:59 PM

To reproduce the crash, I just start a transfer from a UNC NAS to a local FTP server. It is comparing with delete about 5,000 files. If it finishes, it takes about 20 seconds. Normally it crashes in the first 5 seconds (if the debug software isn't running). I'll keep trying to get a good capture. Here is the Microsoft information:
Problem signature:
Problem Event Name: APPCRASH
Application Name: SmartFTP.exe
Application Version: 4.0.1148.0
Application Timestamp: 4cfeb65d
Fault Module Name: sfTransferQueue.dll
Fault Module Version: 4.0.1148.0
Fault Module Timestamp: 4cfeb3ad
Exception Code: c0000005
Exception Offset: 0000000000085088
OS Version: 6.0.6002.2.2.0.272.7
Locale ID: 1033
Additional Information 1: 5d25
Additional Information 2: f0fe656e9b5f1aa149a043971a2c5950
Additional Information 3: ac05
Additional Information 4: e127402975c457dd023d2e0ee38de0ac

Read our privacy statement:
http://go.microsoft....63&clcid=0x0409

#14 mb

mb

    Developer

  • Administrators
  • 11527 posts
  • Gender:
    Male
  • Location:
    Worldwide

Posted 16 December 2010 - 08:16 PM

We need more information:
Does it matter if it is a transfer from a UNC path or not?
Does it matter if you set the Synchronization to One-Way with Delete or just to One-Way?
How many files do we need to make it crash? Does it happen with just one file? e.g. try a transfer of one file a couple of times.
Does the number of workers have any influence? E.g. try it with 1 worker.

#15 CraigBee

CraigBee
  • Members
  • 19 posts

Posted 16 December 2010 - 09:12 PM

Does it matter if it is a transfer from a UNC path or not?
This is actually harder to test than it would seem. I'll get to this tomorrow.

Does it matter if you set the Synchronization to One-Way with Delete or just to One-Way?
It doesn't matter. It doesn't even matter if a file actually transfers. It is crashing even when the directories are identical and it is just running through a directory comparison.

How many files do we need to make it crash? Does it happen with just one file? e.g. try a transfer of one file a couple of times.
I can't get it to crash with just one file. It will not complete the full ~5,000 files when using the local FTP server. It will usually complete the full ~5,000 files when using one of our CDN FTP servers.

Does the number of workers have any influence? E.g. try it with 1 worker.
With 1 worker it takes about 50x longer to crash then with 10 or 20 workers. With 10 or 20 workers it will crash in the first 5 seconds. With 1 worker, it took about 45 minutes to crash.

No crashes:
There are no scenarios I can get SmartFTP to crash using the 32-bit build 1145 on a Windows XP laptop.
There are no scenarios I can get SmartFTP to crash using the 64-bit build 1145/1148/1149 when using the debugger on Windows 2008 server x64. I've been trying this for hours now.

Crash using x64 on Windows 2008 server x64 on a VMWare virtual machine:
It crashes in the first 5 seconds when using a local FTP server and 10 or 20 workers. It takes about 30 to 60 minutes to crash with 1 worker thread. I will crash after a much longer period connected to any of the three CDN FTP servers with 10 or 20 workers.
It is random when it crashes, I can't seem to find any specific trigger.
Bandwidth limits to not have an impact on when it will crash.

Next Steps:
I'm going to let it run over night with the debug on. Its very very slow, so maybe it will crash eventually. I'll test file uploads from the local drive tomorrow as well. I'm sorry I'm not able to get very good debug information. I appreciate you working on this using the limited information I am able to provide.

I tried to install the 32-bit version on the Windows 2008 server, but only the 64-bit version will install, so I can't test the 32-bit version on the server. The problem seems specific to this one combination of configurations.

#16 mb

mb

    Developer

  • Administrators
  • 11527 posts
  • Gender:
    Male
  • Location:
    Worldwide

Posted 16 December 2010 - 09:51 PM

Thank you for collecting all this information. It looks like a typical thread concurrency issue (missing critical section) which happens more often with a higher number of concurrent threads (e.g. more transfer queue workers) and "faster code execution" (no debugger attached).

Microsoft provides another tool than the recommended advplus to collect crash dumps:
http://www.microsoft...&displaylang=en
It might be worth a try to see whether it is less "intrusive" than advplus which as a result would trigger the crash faster. But I have a feeling it might actually do the same as advplus under the hood.

I will review the changes in the transfer queue that has been made between build 1045 and 1047 and see if I can spot the bug. But the best way is always a crash dump :-)

#17 CraigBee

CraigBee
  • Members
  • 19 posts

Posted 20 December 2010 - 02:18 PM

Build 1152 solved the problem. Thanks!

#18 CraigBee

CraigBee
  • Members
  • 19 posts

Posted 11 January 2011 - 06:25 PM

To correct the above post, 1151 fixed the problem, and that is what I installed. I noticed the monitor folder wasn't working, so I upgraded to build 1155. Build 1155 just crashed. I'll test a little more.

#19 CraigBee

CraigBee
  • Members
  • 19 posts

Posted 17 January 2011 - 10:39 PM

The crash must have been a fluke or our virtual machine. SmartFTP has been trouble free and stable for 7 days running 7/24. Awesome!

Craig

#20 mb

mb

    Developer

  • Administrators
  • 11527 posts
  • Gender:
    Male
  • Location:
    Worldwide

Posted 17 January 2011 - 10:51 PM

To get the dump of a sporadic crash we recommend this method:
http://www.smartftp....udio-f2645.html

The benefit is that Tthe debugger will only be started after the application has crashed and won't use any resources while the application is running.




0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users