Slow directory listing with 2000+ files

Stevezau
Posts: 13
Joined: Tue Sep 16, 2008 1:28 am

Slow directory listing with 2000+ files

Post by Stevezau » Thu May 26, 2011 12:56 am

Hi,

Some of the folders i have have 1000's of sub folders/files..

I like to sort by date but they causes SmartFTP to go to 100% cpu usage for ages (5mins+) for it to sort the list..

Is there anyway to speed this up?

I'm using the latest stable version and have a Lenovo w520 laptop which has an Intel i7 CPU (8cores) w/ Win7 SP1.

Thanks
Steve

mb
Posts: 12030
Joined: Thu Oct 11, 2001 1:24 pm
Display name: Mat (SmartFTP)
Contact:

Re: Slow directory listing with 2000+ files

Post by mb » Thu May 26, 2011 1:45 pm

What is the exact total number of files in the directory?

mb
Posts: 12030
Joined: Thu Oct 11, 2001 1:24 pm
Display name: Mat (SmartFTP)
Contact:

Re: Slow directory listing with 2000+ files

Post by mb » Thu May 26, 2011 3:27 pm

I tested the following:
- Directory with 100'000 files
- Takes about 10 seconds to list the first time. 50-70% cpu usage on 1 core of 8.
- Sort takes 5 seconds on any column. ~100% cpu usage on 1 core of 8.

Operating system: Windows 7 SP1 64-bit. CPU: i7 2720qm
Last edited by mb on Thu May 26, 2011 3:59 pm, edited 1 time in total.

Jess Peebles
Posts: 2
Joined: Tue May 31, 2011 11:50 pm

Re: Slow directory listing with 2000+ files

Post by Jess Peebles » Wed Jun 01, 2011 12:02 am

Hi,
We are experiencing the same issue.

We have folders with circa 6000 files.
When we open the directory with any recent version, the contents of the folder will take approx 2 minutes to display, during this time SMART FTP consumes a lot of CPU.

We have tested previous versions, and this issue was introduced after version 4.0.1072.0.
Rolling back to version 4.0.1072.0 and then opening the same directory takes appox 3 seconds to retrieve and display with very little resource.

We are using Lenovo Laptops all running Windows XP.
We can replicate on our T61, T400, and T410 model

Stevezau
Posts: 13
Joined: Tue Sep 16, 2008 1:28 am

Re: Slow directory listing with 2000+ files

Post by Stevezau » Wed Jun 01, 2011 12:09 am

Sorry for the late reply, was on holidays.

I have a few folders that contain around 4000 folders in each (not files).

The listing is taking ages.

Using lenovo w520 laptop.

Anyway i can do some troubleshooting??

PS: I should add that my default sorting is by the date column.
Last edited by Stevezau on Wed Jun 01, 2011 12:09 am, edited 1 time in total.

mb
Posts: 12030
Joined: Thu Oct 11, 2001 1:24 pm
Display name: Mat (SmartFTP)
Contact:

Re: Slow directory listing with 2000+ files

Post by mb » Wed Jun 01, 2011 12:10 am

>Jess Peeble
On Pre-Windows Vista operating system we are using several workarounds to emulate features that are only available in newer operating systems. This causes the performance hit on these legacy operating systems (namely Windows XP and Windows 2003). One way to improve the performance is to install Windows Search on Windows XP / Windows 2003 from here: http://www.microsoft.com/downloads/en/d ... laylang=en

On Windows Vista / Windows 2008 or higher there are no known issues and we are unable to reproduce the problem the starter of this thread has reported.

mb
Posts: 12030
Joined: Thu Oct 11, 2001 1:24 pm
Display name: Mat (SmartFTP)
Contact:

Re: Slow directory listing with 2000+ files

Post by mb » Wed Jun 01, 2011 12:11 am

>Stevezau
Do you also see the problem with folders containing a lot of files but few folders?

Stevezau
Posts: 13
Joined: Tue Sep 16, 2008 1:28 am

Re: Slow directory listing with 2000+ files

Post by Stevezau » Wed Jun 01, 2011 12:31 am

mb,

I created a new folder with 7000 files and it was very quick..

I then created another folder (for testing purposes) with 7000 sub folders and it is taking much longer to sort but strangely enough my other folders which have 3000-4000 sub folders (each sub folder has a different modification date and size) are taking longer?

Stevezau
Posts: 13
Joined: Tue Sep 16, 2008 1:28 am

Re: Slow directory listing with 2000+ files

Post by Stevezau » Wed Jun 01, 2011 12:32 am

I've just noticed that it appears right after it downloads the listing the folders will appear and I can scroll for 1-2 seconds then smartftp freezes up... Maybe it's sorting them by date in the background or something?

mb
Posts: 12030
Joined: Thu Oct 11, 2001 1:24 pm
Display name: Mat (SmartFTP)
Contact:

Re: Slow directory listing with 2000+ files

Post by mb » Wed Jun 01, 2011 12:55 am

>Stevezau
Correct. Sorting is done in the background. I will try to reproduce it with 100k folders (instead of files). Thanks for testing.

Locked