Bulk Delete of Files / Folders

We are currently developing a new application with SmartFTP library and we are seeing a problem that we cannot work out.

Our Application uploads a whole directory tree and it appears to have worked correctly. At some point later in the day these files need to be deleted via FTP so we have a function that recursively builds a list of all the files in the directories and then uses the Delete File function checking for FTPerror.FtpErrorSuccess. We are consistently seeing Errors in the delete function indicating that the File we are processing (strFile in the example code below) doesn’t exist, yet we have only just got a list of the files from the remote server. We also see FTPError {15} occasionally. In our test and live environments we are using Gene6 FTP V3.10.

DELETE FUNCTION
For each strFile in collFiles
If Not objFTPConn.DeleteFile(strFile) = enumError.ftpErrorSuccess Then
MsgBox("Unable to Delete File " & strFile)
GenericServices.WriteToLog(strFile, "Delete Failure", Now() - Now(), TransferType.Delete, 0, Me)
Else
collFiles.Remove(strFile)
End If
Next

We have noticed that during the upload of these files the SmartFTP connection seems to momentarily drop the connection and it is on these files that we have the error where the FTPListing says they exist yet when we delete them they appear not to.

When we come to delete the files and their respective folders later in the day, we get an error indicating that the directory is not empty yet no files exist except in the SmartFTP component. Can you shed some light on this possible error please?

Set the LogFile property of the IFTPConnection interface to a log file and then inspect it.

Regards,
Mat

Do you have a log which shows the error when deleting files?

If the directory delete fails please create a function to read this directory and output its content into your own log. Then you will see what files are left in the directory.

Regards,
Mat

Do you have a log which shows the error when deleting files?

If the directory delete fails please create a function to read this directory and output its content into your own log. Then you will see what files are left in the directory.

Regards,
Mat

Mat,

We have done this and we can see 5-6 files in each directory that fails. The files that appear to exist are the same file names where we have had the 'Abort' Error detailed in my earlier posting.

Our business process is...
1. Retrieve File List from Server
2. Process Files and Mark record in SQL Server indicating process complete
3. Build List of Files on Server to Delete
4. Delete Files
5. Build List of Directories
6. Delete Directories

In reality 3,4,5 and 6 above are;
After processing them we retrieve a SfFTPLib listing from the server and itterate through it (multiple directories) and delete each file in the collection (sfFTPlib). This appears to work. Only when we have deleted all these files do we then process a list of the directories and remove each one of them in turn. When we try this we get the Unable to Remove Directory as it is not empty. If we re-do the list files then we see the list of files remaining in the directory.

In the live environment, there will be upto 8 branch offices uploading 2000-3000 files each per day in a directory structure similar to;

{date}YYYYMMMDD
{Region}
{CustomerIdentifier}
{Contracts}
{SubContracts}


Typically we have several Sub Contracts within a Contract, linkd to a Customer who may have written several contracts on that day. Each Branch office covers several geogrphical regions

Hello ...

Maybe your offices are uploading files while your application is between step 3-6.

Regards,
Mat

Hello ...

Maybe your offices are uploading files while your application is between step 3-6.

Regards,
Mat

This system is currently in test and therefore not rolled out to the offices yet, we do have a limited user trial running, this is where we first noticed the issue. We now replicate the whole scenario to our Test environment. (I detailed the process so you can gain a better understndig of what we are doing, I have also said we are developing... etc).

I note with interest that you have not commented on the ABORT in the key exchange with the Gene6 FTP server. Having been through the application code and 16MB of logs, we see this error consistently and it appears to be the root of the problem we are experiencing.

WE feel that there may an issue with the SMartFTP Key exchange under certain circumstances which causes the abort and hence leads to our issue. I appreciate that our circumstances are very specific to us and we thank you for your help.

Syd

Hello Syd ..

>ABOR problem
I need to be able to reproduce the problem with key exchange to debug it. From the log it looks like the server disconnects the connection and not the client. Maybe you find something in the server log.

Regardless of reason of the ABOR problem your application should be able to handle such cases. It means it should always verify if the file has been successfully uploaded. If the upload fails you have to use the SIZE command to check if the file exists on the server. In this case you delete the file immediately.

Regards,
Mat

Hello Syd ..

>ABOR problem
I need to be able to reproduce the problem with key exchange to debug it. From the log it looks like the server disconnects the connection and not the client. Maybe you find something in the server log.

Regardless of reason of the ABOR problem your application should be able to handle such cases. It means it should always verify if the file has been successfully uploaded. If the upload fails you have to use the SIZE command to check if the file exists on the server. In this case you delete the file immediately.

Regards,
Mat

Hey Ho Mat,

Yes we already handle all the exceptions in our own error handling code and we check uploads exactly the way you suggest. We also handle the issues with the connection, but as you can see from the Log posted earlier we see the ABOR in the Key Exchange - We do not see any errors that we can trap on the return.

When you perform a ReadDirectory() and assign to sfFTPLib.Items object we have a sfFTPLib.Items collection that doesn't include the failed uploaded file (this is correct as we see it and our code handles the removal of any failed uploads should they occur), but when we come to delete the directory in which it should have been uploaded we find that the FTP Server still thinks the file exists there.

One of our developers mentioned that he thought that the ABOR happened after approximately every 200 File Uploads, but we have yet to document this on our Test environment.

I have now managed to prepare 2 Logs..... 1 from the Smart FTP Client and the other from GENE6 FTP. They show the same timeframe for the job. It may help identify the ABOR (if indeed it is an issue). How do I get them to you?

I look forward to your reply.

Syd

You can email them to support at smartftp.com the detailed error information should be displayed in the log from gene6 since it disconnects the data connection.

Regards,
Mat