Lengthy directory transfer and modify time failure

moxland

2006-05-31 01:07:40

I am trying to download a web site using the transfer queue. I simply selected the contents of the remote site root folder and added it to the queue and started it. Each file seems to first download the directory listing of the remote directory containing the file, before downloading the file. There are several directories in the site that contain thousands of files, and one very large directory listing in the site is 3mb. It is transferring 3.02mb to transfer a 20k image in that directory. So, it takes much much longer to transfer a file from a directory containing more files. Also, transfer speed is fine between this host and my destination, usually around 60k/s. Downloading the directory listing before each file is causing the download of this 2GB site to take days and possibly weeks instead of hours. Is there any way to avoid having to download a directory listing before each file?

Also, none of the files are transferring their dates and times. When I directly transfer a file, the overwrite confirmation window that pops up shows the new file's date and time as "Unknown" and it is stored as the year 1601 on the Windows destination. Is there a reason why the date and time would not be transferring? Perhaps having something to do with the MDTM command failing? Dates and times have transferred successfully with other FTP clients.

The log of one file's queue transfer is as follows:

[20:45:04] SIZE 400000004505.gif
[20:45:04] 213 67672
[20:45:04] MDTM 400000004505.gif
[20:45:04] 550 '400000004505.gif' : no such file or directory.
[20:45:04] TYPE A
[20:45:04] 200 Type set to A.
[20:45:04] PASV
[20:45:04] 227 Entering Passive Mode (12,33,128,123,205,181).
[20:45:04] Opening data connection to 12.33.128.123 Port: 52661
[20:45:04] LIST -aL
[20:45:09] 150 Data connection accepted from 208.101.19.165:3187; transfer starting.
[20:47:19] 226 Transfer ok
[20:47:19] 2935443 bytes transferred. (21.3 KB/s) (00:02:14)
[20:47:19] TYPE I
[20:47:19] 200 Type set to I.
[20:47:19] PASV
[20:47:19] 227 Entering Passive Mode (12,33,128,123,27,93).
[20:47:19] Opening data connection to 12.33.128.123 Port: 7005
[20:47:19] RETR 400000004505.gif
[20:47:19] 150 Data connection accepted from 208.101.19.165:3206; transfer starting for 400000004505.gif (67672 bytes).
[20:47:21] 226 Transfer ok
[20:47:21] 67672 bytes transferred. (47.8 KB/s) (00:00:01)
[20:47:21] MDTM 400000004505.gif
[20:47:21] 550 '400000004505.gif' : no such file or directory.

Thank you very much!
Mox

mb

2006-05-31 04:38:24

What version of SmartFTP are you using? Please post the system information from the menu: Help->About "System Information" dialog.

>MDTM
Your FTP server doesn't support the feature to set the time of the remote file. Therefore the client (SmartFTP) has no control over the file time of the uploaded files.

Thank you.
Regards,
SmartFTP

moxland

2006-05-31 16:25:37

Contents of the System Info tab in the About dialog:

+- System -----------------------------
Microsoft Windows Server 2003 Standard Edition
Service Pack 1 (Build 3790)

CPU Speed : 3002 MHz
Total Memory : 327144 KB
Free Memory : 47252 KB

+- SmartFTP ---------------------------
Version : 2.0.996.17
Time Stamp : 2006-04-30 04:56:59
Platform : x86

+- Application DLL --------------------
sfFTPLib.dll : 1.5.8.5
SmartHook.dll : 1.0.2.1

+- Language ---------------------------
SmartFTP.exe : 2.0.996.17

+- System DLL -------------------------
shell32.dll : 6.00.3790.2662 (srv03_sp1_gdr.060316-1532)
shlwapi.dll : 6.00.3790.2653 (srv03_sp1_gdr.060303-1536)
comctl32.dll : 6.0 (srv03_sp1_rtm.050324-1447)
riched20.dll : 5.31.23.1224
schannel.dll : 5.2.3790.1830 (srv03_sp1_rtm.050324-1447)

+- Internet Explorer ------------------
Version : 6.0.3790.1830

+- Winsock ----------------------------
Winsock : 2.2


>Your FTP server doesn't support the feature to set the time of the remote file. Therefore the client (SmartFTP) has no control over the file time of the uploaded files.

I am not trying to upload files, I am simply trying to download all the files of a web site to the site's new host. I am operating SmartFtp through RemoteDesktop on the actual destination host, so it is simply a download. However, when the files are copied to the destination, they are saved without dates. There must be a way to get the date/times associated with each download file and save it with each destination file. The date/times are listed correctly under the Date column in the source host's file list, so I don't see why an FTP client can't access those date/times to save with the downloaded files. Other FTP clients I have used have at least saved the current date instead of leaving the date empty.

Thanks

moxland

2006-06-02 16:32:15

Please respond to my two questions. My client is waiting on the transferred site. I purchased a SmartFtp license and I would like a little help, please.

mb

2006-06-02 16:40:16

The directory listings are caused because your server doesn't return anything meaningful in the MDTM reply. It claims to support MDTM but in reality it's broken or it doesn't work as expected. If you disable "Keep File Time" for downloads you shouldn't see the additional directory listings. All the file dates of the downloaded files will be set to the current date then.

Regards,
SmartFTP

moxland

2006-06-03 01:05:22

Thank you for your response. I unchecked all three of the Keep File Time checkboxes, including the Download one. The files are now being saved with today's date and time, which is fine. However, the one remaining problem is that each file transfer still gets that file's potentially huge directory listing. This happens in both direct and queue mode.

I do not think it has changed, but the log for transferring one image file is now this:

[19:45:19] SIZE 100000000002.jpg
[19:45:19] 213 19319
[19:45:19] MDTM 100000000002.jpg
[19:45:19] 550 '100000000002.jpg' : no such file or directory.
[19:45:19] TYPE A
[19:45:19] 200 Type set to A.
[19:45:19] PASV
[19:45:19] 227 Entering Passive Mode (12,33,128,123,91,152).
[19:45:20] Opening data connection to 12.33.128.123 Port: 23448
[19:45:20] LIST -aL
[19:45:24] 150 Data connection accepted from 208.101.19.165:3973; transfer starting.
[19:46:13] 226 Transfer ok
[19:46:13] 2942013 bytes transferred. (53.5 KB/s) (00:00:53)
[19:46:13] TYPE I
[19:46:13] 200 Type set to I.
[19:46:13] PASV
[19:46:13] 227 Entering Passive Mode (12,33,128,123,63,239).
[19:46:13] Opening data connection to 12.33.128.123 Port: 16367
[19:46:13] RETR 100000000002.jpg
[19:46:14] 150 Data connection accepted from 208.101.19.165:3978; transfer starting for 100000000002.jpg (19319 bytes).
[19:46:14] 226 Transfer ok
[19:46:14] 19319 bytes transferred. (26.9 KB/s) (701 ms)

Don't know what to do next. Transferring a directory of 40,000 files takes about 20 seconds each file because I have to download the 3mb directory listing each time at about 50k/s. Transferring the directory is going to take about 9 days! It should only take a few hours if the directory listing wasn't being requested each time. Please help.

Thank you,
Mox

mb

2006-06-03 01:17:00

Hello ..

I will take a look at this. Do you use the Transfer Queue for these downloads?

Regards,
SmartFTP

moxland

2006-06-03 12:18:07

Yes, this log was made using the transfer queue, which is what I prefer to use. The excessive directory listing is also happening in direct mode, even with "Keep file time" unchecked.

Thanks!