Jump to content


Photo

Bulk Delete of Files / Folders


  • Please log in to reply
9 replies to this topic

#1 UncleSyd

UncleSyd
  • Members
  • 19 posts
  • Location:
    Wales

Posted 29 July 2008 - 10:59 AM

We are currently developing a new application with SmartFTP library and we are seeing a problem that we cannot work out.

Our Application uploads a whole directory tree and it appears to have worked correctly. At some point later in the day these files need to be deleted via FTP so we have a function that recursively builds a list of all the files in the directories and then uses the Delete File function checking for FTPerror.FtpErrorSuccess. We are consistently seeing Errors in the delete function indicating that the File we are processing (strFile in the example code below) doesn’t exist, yet we have only just got a list of the files from the remote server. We also see FTPError {15} occasionally. In our test and live environments we are using Gene6 FTP V3.10.

DELETE FUNCTION
For each strFile in collFiles
If Not objFTPConn.DeleteFile(strFile) = enumError.ftpErrorSuccess Then
MsgBox("Unable to Delete File " & strFile)
GenericServices.WriteToLog(strFile, "Delete Failure", Now() - Now(), TransferType.Delete, 0, Me)
Else
collFiles.Remove(strFile)
End If
Next

We have noticed that during the upload of these files the SmartFTP connection seems to momentarily drop the connection and it is on these files that we have the error where the FTPListing says they exist yet when we delete them they appear not to.

When we come to delete the files and their respective folders later in the day, we get an error indicating that the directory is not empty yet no files exist except in the SmartFTP component. Can you shed some light on this possible error please?

#2 mb

mb

    Developer

  • Administrators
  • 11521 posts
  • Gender:
    Male
  • Location:
    Worldwide

Posted 29 July 2008 - 03:06 PM

Set the LogFile property of the IFTPConnection interface to a log file and then inspect it.

Regards,
Mat

#3 UncleSyd

UncleSyd
  • Members
  • 19 posts
  • Location:
    Wales

Posted 29 July 2008 - 04:42 PM

Set the LogFile property of the IFTPConnection interface to a log file and then inspect it.

Regards,
Mat

Mat,

Thanks, we had already examined the logs before raising the issue with you. Here is a section of the log that is relevant... you can see that the Conenction gets aborted during the STOR command and we never get a confirmation of a successful file upload. I can only assume that the SmartFTP library recognises the drop and reconnects automatically and returns a failure to the SmartFTP.UploadFile call. Our code watches for the failure and if so, part of it's clean up is to attempt to delete the file that was just uploaded. We've also tried NOT doing a clean-up operation but that produces the same effect.

What we see after the upload is;

When you perform a ReadDirectory() and assign to sfFTPLib.Items object we have a sfFTPLib.Items collection that doesn't include the failed uploaded file (this is correct as we see it), but when we come to delete the directory in which it should have been uploaded we find that it still thinks the file exists there.

Curious Huh?

Syd

[20080729 15:47:08] 226-Compression : 100% (8 Bytes/0 Bytes).
[20080729 15:47:08] 226 Transfer ok.
[20080729 15:47:08] PASV
[20080729 15:47:08] 227 Entering Passive Mode (192,168,100,12,125,252)
[20080729 15:47:08] Opening data connection to 192.168.100.12 Port: 32252
[20080729 15:47:08] STOR \ICONS\BMP\Regular\16x16\Filter Feather.bmp.822.1.128582402700000000
[20080729 15:47:08] Connected. Exchanging encryption keys...
[20080729 15:47:08] An established connection was aborted by the software in your host machine.
[20080729 15:47:08] Server closed connection
[20080729 15:47:08] SmartFTP FTP Library v1.5.13.35
[20080729 15:47:08] Resolving host name "192.168.100.12"
[20080729 15:47:08] Connecting to 192.168.100.12 Port: 21
[20080729 15:47:08] Connected to 192.168.100.12.
[20080729 15:47:08] 220 Gene6 FTP Server v3.10.0 (Build 2) ready...
[20080729 15:47:08] AUTH TLS
[20080729 15:47:08] 234 AUTH command ok; starting SSL connection.
[20080729 15:47:08] Connected. Exchanging encryption keys...
[20080729 15:47:08] Session Cipher: 128 bit AES
[20080729 15:47:08] TLS encrypted session established.
[20080729 15:47:08] PBSZ 0
[20080729 15:47:08] 200 PBSZ=0
[20080729 15:47:08] USER KP901000279
[20080729 15:47:08] 331 Password required for KP901000279.
[20080729 15:47:08] PASS (hidden)
[20080729 15:47:08] 230 User KP901000279 logged in.
[20080729 15:47:08] SYST
[20080729 15:47:08] 215 UNIX Type: L8
[20080729 15:47:08] Detected Server Type: UNIX
[20080729 15:47:08] FEAT
[20080729 15:47:08] 211-Extensions supported:
[20080729 15:47:08] AUTH TLS
[20080729 15:47:08] CCC
[20080729 15:47:08] CLNT
[20080729 15:47:08] CPSV
[20080729 15:47:08] EPRT
[20080729 15:47:08] EPSV
[20080729 15:47:08] MDTM
[20080729 15:47:08] MFCT
[20080729 15:47:08] MFMT
[20080729 15:47:08] MLST type*;size*;create;modify*;
[20080729 15:47:08] MODE Z
[20080729 15:47:08] PASV
[20080729 15:47:08] PBSZ
[20080729 15:47:08] PROT
[20080729 15:47:08] REST STREAM
[20080729 15:47:08] SIZE
[20080729 15:47:08] SSCN
[20080729 15:47:08] TVFS
[20080729 15:47:08] UTF8
[20080729 15:47:08] XCRC "filename" SP EP
[20080729 15:47:08] XMD5 "filename" SP EP
[20080729 15:47:08] XSHA1 "filename" SP EP
[20080729 15:47:08] 211 End.
[20080729 15:47:08] CLNT CEREDIR1
[20080729 15:47:08] 200 Noted.
[20080729 15:47:08] Detected Server Software: Gene6 FTP Server 3.x
[20080729 15:47:08] OPTS UTF8 ON
[20080729 15:47:08] 200 UTF8 OPTS ON
[20080729 15:47:08] PWD
[20080729 15:47:08] 257 "/" is current directory.
[20080729 15:47:08] DELE \ICONS\BMP\Regular\16x16\Filter Feather.bmp.822.1.128582402700000000
[20080729 15:47:08] 250 File "/ICONS/BMP/Regular/16x16/Filter Feather.bmp.822.1.128582402700000000" deleted.
[20080729 15:47:08] NOOP
[20080729 15:47:08] 200 NOOP command successful.
[20080729 15:47:09] NOOP

#4 mb

mb

    Developer

  • Administrators
  • 11521 posts
  • Gender:
    Male
  • Location:
    Worldwide

Posted 29 July 2008 - 05:15 PM

Do you have a log which shows the error when deleting files?

If the directory delete fails please create a function to read this directory and output its content into your own log. Then you will see what files are left in the directory.

Regards,
Mat

#5 UncleSyd

UncleSyd
  • Members
  • 19 posts
  • Location:
    Wales

Posted 29 July 2008 - 11:25 PM

Do you have a log which shows the error when deleting files?

If the directory delete fails please create a function to read this directory and output its content into your own log. Then you will see what files are left in the directory.

Regards,
Mat

Mat,

We have done this and we can see 5-6 files in each directory that fails. The files that appear to exist are the same file names where we have had the 'Abort' Error detailed in my earlier posting.

Our business process is...
1. Retrieve File List from Server
2. Process Files and Mark record in SQL Server indicating process complete
3. Build List of Files on Server to Delete
4. Delete Files
5. Build List of Directories
6. Delete Directories

In reality 3,4,5 and 6 above are;
After processing them we retrieve a SfFTPLib listing from the server and itterate through it (multiple directories) and delete each file in the collection (sfFTPlib). This appears to work. Only when we have deleted all these files do we then process a list of the directories and remove each one of them in turn. When we try this we get the Unable to Remove Directory as it is not empty. If we re-do the list files then we see the list of files remaining in the directory.

In the live environment, there will be upto 8 branch offices uploading 2000-3000 files each per day in a directory structure similar to;

{date}YYYYMMMDD
{Region}
{CustomerIdentifier}
{Contracts}
{SubContracts}


Typically we have several Sub Contracts within a Contract, linkd to a Customer who may have written several contracts on that day. Each Branch office covers several geogrphical regions

#6 mb

mb

    Developer

  • Administrators
  • 11521 posts
  • Gender:
    Male
  • Location:
    Worldwide

Posted 29 July 2008 - 11:33 PM

Hello ...

Maybe your offices are uploading files while your application is between step 3-6.

Regards,
Mat

#7 UncleSyd

UncleSyd
  • Members
  • 19 posts
  • Location:
    Wales

Posted 30 July 2008 - 06:39 AM

Hello ...

Maybe your offices are uploading files while your application is between step 3-6.

Regards,
Mat

This system is currently in test and therefore not rolled out to the offices yet, we do have a limited user trial running, this is where we first noticed the issue. We now replicate the whole scenario to our Test environment. (I detailed the process so you can gain a better understndig of what we are doing, I have also said we are developing... etc).

I note with interest that you have not commented on the ABORT in the key exchange with the Gene6 FTP server. Having been through the application code and 16MB of logs, we see this error consistently and it appears to be the root of the problem we are experiencing.

WE feel that there may an issue with the SMartFTP Key exchange under certain circumstances which causes the abort and hence leads to our issue. I appreciate that our circumstances are very specific to us and we thank you for your help.

Syd

#8 mb

mb

    Developer

  • Administrators
  • 11521 posts
  • Gender:
    Male
  • Location:
    Worldwide

Posted 30 July 2008 - 06:57 AM

Hello Syd ..

>ABOR problem
I need to be able to reproduce the problem with key exchange to debug it. From the log it looks like the server disconnects the connection and not the client. Maybe you find something in the server log.

Regardless of reason of the ABOR problem your application should be able to handle such cases. It means it should always verify if the file has been successfully uploaded. If the upload fails you have to use the SIZE command to check if the file exists on the server. In this case you delete the file immediately.

Regards,
Mat

#9 UncleSyd

UncleSyd
  • Members
  • 19 posts
  • Location:
    Wales

Posted 30 July 2008 - 04:58 PM

Hello Syd ..

>ABOR problem
I need to be able to reproduce the problem with key exchange to debug it. From the log it looks like the server disconnects the connection and not the client. Maybe you find something in the server log.

Regardless of reason of the ABOR problem your application should be able to handle such cases. It means it should always verify if the file has been successfully uploaded. If the upload fails you have to use the SIZE command to check if the file exists on the server. In this case you delete the file immediately.

Regards,
Mat

Hey Ho Mat,

Yes we already handle all the exceptions in our own error handling code and we check uploads exactly the way you suggest. We also handle the issues with the connection, but as you can see from the Log posted earlier we see the ABOR in the Key Exchange - We do not see any errors that we can trap on the return.

When you perform a ReadDirectory() and assign to sfFTPLib.Items object we have a sfFTPLib.Items collection that doesn't include the failed uploaded file (this is correct as we see it and our code handles the removal of any failed uploads should they occur), but when we come to delete the directory in which it should have been uploaded we find that the FTP Server still thinks the file exists there.

One of our developers mentioned that he thought that the ABOR happened after approximately every 200 File Uploads, but we have yet to document this on our Test environment.

I have now managed to prepare 2 Logs..... 1 from the Smart FTP Client and the other from GENE6 FTP. They show the same timeframe for the job. It may help identify the ABOR (if indeed it is an issue). How do I get them to you?

I look forward to your reply.

Syd

Edited by UncleSyd, 30 July 2008 - 05:28 PM.


#10 mb

mb

    Developer

  • Administrators
  • 11521 posts
  • Gender:
    Male
  • Location:
    Worldwide

Posted 30 July 2008 - 09:42 PM

You can email them to support at smartftp.com the detailed error information should be displayed in the log from gene6 since it disconnects the data connection.

Regards,
Mat




0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users