This is taking forevvvvvvver....

This forum is not monitored by the SmartFTP staff. To contact technical support see KB: How to request technical support.
SamOldman
Posts: 8
Joined: Sat Jun 28, 2014 9:42 pm

This is taking forevvvvvvver....

Post by SamOldman » Sat Jun 28, 2014 9:49 pm

First up - I like SmartFTP.  I really do...  but it does some things that really really drive me crazy.
 
I'm trying to upload somewhere's in the neighborhood of 15,000 .jpg files, all of which are around 40KB in size.
 
Unfortunately SmartFTP insists on using the MLST feature to list the specs of each file - ostensibly for File Exist checking.  The problem is that I cannot disable the MLST announcement, and SmartFTP seems incapable of ignoring it.  All of this means that I have to sit here for a good 30 minutes or so watching this idiotic system send 15,000 MLST commands to my server before uploading any files.
 
So for refresher : 
  •  I cannot disable the MLST broadcast from the feature list.
  •  I cannot stop SmartFTP from sending an MLST for each file before uploading.
 
I'd REALLY appreciate it if someone could tell me how to upload WITHOUT SmartFTP trying to send the MLST command.
 

SamOldman
Posts: 8
Joined: Sat Jun 28, 2014 9:42 pm

Re: This is taking forevvvvvvver....

Post by SamOldman » Tue Jul 01, 2014 9:30 pm

Just had to sit here for 1 hour and 21 minutes while SmartFTP sent MLST requests, trying to upload files.

mb
Posts: 12026
Joined: Thu Oct 11, 2001 1:24 pm
Display name: Mat (SmartFTP)
Contact:

Re: This is taking forevvvvvvver....

Post by mb » Wed Jul 02, 2014 12:20 am

We are looking into the problem.

mb
Posts: 12026
Joined: Thu Oct 11, 2001 1:24 pm
Display name: Mat (SmartFTP)
Contact:

Re: This is taking forevvvvvvver....

Post by mb » Wed Jul 02, 2014 12:21 pm

Can you please post the log from 1-2 file transfers?

SamOldman
Posts: 8
Joined: Sat Jun 28, 2014 9:42 pm

Re: This is taking forevvvvvvver....

Post by SamOldman » Wed Jul 02, 2014 3:30 pm

Sure thing.  I had a small upload (only around 6,000 files) this morning so I might as well post that log.  This took around 8 minutes *just to process* before beginning the upload.  Some days I have 100,000 files to upload.  Those days suck.
 
Anyway - here's the log, which I have censored to hide personal information :
 
[11:12:38] SmartFTP v6.0.2023.0
[11:12:38] 1>Resolving host name "XXX.XXX.XXX.XXX"
[11:12:38] 1>Connecting to XXX.XXX.XXX.XXX Port: 21
[11:12:38] 1>Connected to XXX.XXX.XXX.XXX.
[11:12:38] 1>220---------- Welcome to Pure-FTPd [privsep] [TLS] ----------
[11:12:38] 1>220-You are user number 21 of 500 allowed.
[11:12:38] 1>220-Local time is now 08:14. Server port: 21.
[11:12:38] 1>220-This is a private system - No anonymous login
[11:12:38] 1>220 You will be disconnected after 3 minutes of inactivity.
[11:12:38] 1>USER XXXXXXXXXXXX
[11:12:38] 1>331 User XXXXXXXXXXXX OK. Password required
[11:12:38] 1>PASS (hidden)
[11:12:38] 1>230 OK. Current restricted directory is /
[11:12:38] 1>SYST
[11:12:38] 1>215 UNIX Type: L8
[11:12:38] 1>Detected Server Type: UNIX
[11:12:38] 1>RTT: 78.559 ms
[11:12:38] 1>FEAT
[11:12:39] 1>211-Extensions supported:
[11:12:39] 1> EPRT
[11:12:39] 1> IDLE
[11:12:39] 1> MDTM
[11:12:39] 1> SIZE
[11:12:39] 1> MFMT
[11:12:39] 1> REST STREAM
[11:12:39] 1> MLST type*;size*;sizd*;modify*;UNIX.mode*;UNIX.uid*;UNIX.gid*;unique*;
[11:12:39] 1> MLSD
[11:12:39] 1> AUTH TLS
[11:12:39] 1> PBSZ
[11:12:39] 1> PROT
[11:12:39] 1> UTF8
[11:12:39] 1> TVFS
[11:12:39] 1> ESTA
[11:12:39] 1> PASV
[11:12:39] 1> EPSV
[11:12:39] 1> SPSV
[11:12:39] 1> ESTP
[11:12:39] 1>211 End.
[11:12:39] 1>OPTS UTF8 ON
[11:12:39] 1>200 OK, UTF-8 enabled
[11:12:39] 1>PWD
[11:12:39] 1>257 "/" is your current location
[11:12:39] 1>CWD /XXXXXXXX/XXXXXXXX/test
[11:12:39] 1>250 OK. Current directory is /XXXXXXXX/XXXXXXXX/test
[11:12:39] 1>PWD
[11:12:39] 1>257 "/XXXXXXXX/XXXXXXXX/test" is your current location
[11:12:39] 1>TYPE A
[11:12:39] 1>200 TYPE is now ASCII
[11:12:39] 1>PASV
[11:12:39] 1>227 Entering Passive Mode (XXX,XXX,XXX,XXX,195,190)
[11:12:39] 1>Opening data connection to XXX.XXX.XXX.XXX Port: 50110
[11:12:39] 1>LIST -aL
[11:12:39] 1>150 Accepted data connection
[11:12:39] 1>226-Options: -a -l 
[11:12:39] 1>226 2 matches total
[11:12:39] 1>131 bytes transferred. (1.45 KB/s) (88 ms)
[11:13:08] 1>MLST wmdin1b-1k.jpg
[11:13:08] 1>550 Can't check for file existence
[11:13:08] 1>MLST wmdin1b-3r.jpg
[11:13:08] 1>550 Can't check for file existence
[11:13:08] 1>MLST wmdin1b-4rb.jpg
[11:13:08] 1>550 Can't check for file existence
[11:13:08] 1>MLST wmdin1b-5f.jpg
[11:13:08] 1>550 Can't check for file existence
[11:13:08] 1>MLST wmdin1b-14n.jpg
[11:13:08] 1>550 Can't check for file existence
[11:13:08] 1>MLST wmdin1h-1k.jpg
[11:13:09] 1>550 Can't check for file existence
[11:13:09] 1>MLST wmdin1h-3r.jpg
[11:13:09] 1>550 Can't check for file existence
[11:13:09] 1>MLST wmdin1h-4rb.jpg
[11:13:09] 1>550 Can't check for file existence
[11:13:09] 1>MLST wmdin1h-5f.jpg
[11:13:09] 1>550 Can't check for file existence
[11:13:09] 1>MLST wmdin1h-14n.jpg
[11:13:09] 1>550 Can't check for file existence
[11:13:09] 1>MLST wmdin1l-1k.jpg
[11:13:09] 1>550 Can't check for file existence
[11:13:09] 1>MLST wmdin1l-3r.jpg
[11:13:09] 1>550 Can't check for file existence
[11:13:09] 1>MLST wmdin1l-4rb.jpg
[11:13:09] 1>550 Can't check for file existence
 
.
.  This is me skipping ahead since you don't really want to see 12,000 lines of MLST commands which all result in the same reply.
.
 
[11:22:00] 1>MLST wmdin228s-5f.jpg
[11:22:00] 1>550 Can't check for file existence
[11:22:00] 1>MLST wmdin228s-14n.jpg
[11:22:00] 1>550 Can't check for file existence
[11:22:00] 1>MLST wmdin228t-1k.jpg
[11:22:00] 1>550 Can't check for file existence
[11:22:00] 1>MLST wmdin228t-3r.jpg
[11:22:00] 1>550 Can't check for file existence
[11:22:00] 1>MLST wmdin228t-4rb.jpg
[11:22:00] 1>550 Can't check for file existence
[11:22:00] 1>MLST wmdin228t-5f.jpg
[11:22:00] 1>550 Can't check for file existence
[11:22:00] 1>MLST wmdin228t-14n.jpg
[11:22:00] 1>550 Can't check for file existence
[11:22:00] 1>MLST wmdin228ta-1k.jpg
[11:22:01] 1>550 Can't check for file existence
[11:22:01] 1>MLST wmdin228ta-3r.jpg
[11:22:01] 1>550 Can't check for file existence
[11:22:01] 1>MLST wmdin228ta-4rb.jpg
[11:22:01] 1>550 Can't check for file existence
[11:22:01] The operation has been added to the transfer queue. Start the transfer queue to process it.

mb
Posts: 12026
Joined: Thu Oct 11, 2001 1:24 pm
Display name: Mat (SmartFTP)
Contact:

Re: This is taking forevvvvvvver....

Post by mb » Wed Jul 02, 2014 4:47 pm

The new behavior in version 6 will be reverted back in the next build. This should solve the problem you are seeing.
The new build with the changes is available now:
https://www.smartftp.com/download
Last edited by mb on Sun Jul 13, 2014 11:18 pm, edited 1 time in total.

SamOldman
Posts: 8
Joined: Sat Jun 28, 2014 9:42 pm

Re: This is taking forevvvvvvver....

Post by SamOldman » Thu Jul 03, 2014 7:16 pm

Okay, so I closed the remote browser during the batch upload, but I only noticed an increase in speed of around 10%.  It looks like all that did was hide the output from the server.  It's an improvement, but not a significant one.
 
Any other ideas?

mb
Posts: 12026
Joined: Thu Oct 11, 2001 1:24 pm
Display name: Mat (SmartFTP)
Contact:

Re: This is taking forevvvvvvver....

Post by mb » Thu Jul 03, 2014 8:00 pm

No your assumption is wrong.
You can increase the number of workers in the transfer queue.

SamOldman
Posts: 8
Joined: Sat Jun 28, 2014 9:42 pm

Re: This is taking forevvvvvvver....

Post by SamOldman » Fri Jul 04, 2014 1:46 pm

Come on...  Just because I closed the browser window doesn't mean magically nothing happens in the background.  The server still maintains the log, even if I can't see it.  I checked the log on the server after the upload actually began and it showed that SmartFTP still continued with its MLST requests.  The increase in speed I saw could just as easily be attributed to decreased server load.
 
Also, it doesn't matter how many workers I have in the queue because the MLST requests take place BEFORE the transfer begins.  I still have to wait for all of this to complete before the files begin to upload.

Belazor
Posts: 2
Joined: Sun Jun 22, 2008 2:06 pm

Re: This is taking forevvvvvvver....

Post by Belazor » Tue Jul 08, 2014 6:39 pm

I have the same problem, and closing the remote browser does not work.
 
mb wrote: No your assumption is wrong.

You can increase the number of workers in the transfer queue.
 
This is a pretty pathetic answer. Increasing the amount of workers doesn't magically make the server accept more connections per IP, and it doesn't change the fact that SmartFTP v5 did not have this bug. This bug was introduced in v6, and needs to be resolved ASAP.

Post Reply