Jump to content


Photo

Slow directory listing with 2000+ files


This topic has been archived. This means that you cannot reply to this topic.
18 replies to this topic

#1 Stevezau

Stevezau
  • Members
  • 13 posts

Posted 26 May 2011 - 12:56 AM

Hi,

Some of the folders i have have 1000's of sub folders/files..

I like to sort by date but they causes SmartFTP to go to 100% cpu usage for ages (5mins+) for it to sort the list..

Is there anyway to speed this up?

I'm using the latest stable version and have a Lenovo w520 laptop which has an Intel i7 CPU (8cores) w/ Win7 SP1.

Thanks
Steve

#2 mb

mb

    Developer

  • Administrators
  • 11521 posts

Posted 26 May 2011 - 01:45 PM

What is the exact total number of files in the directory?

#3 mb

mb

    Developer

  • Administrators
  • 11521 posts

Posted 26 May 2011 - 03:27 PM

I tested the following:
- Directory with 100'000 files
- Takes about 10 seconds to list the first time. 50-70% cpu usage on 1 core of 8.
- Sort takes 5 seconds on any column. ~100% cpu usage on 1 core of 8.

Operating system: Windows 7 SP1 64-bit. CPU: i7 2720qm

#4 Jess Peebles

Jess Peebles
  • Members
  • 2 posts

Posted 01 June 2011 - 12:02 AM

Hi,
We are experiencing the same issue.

We have folders with circa 6000 files.
When we open the directory with any recent version, the contents of the folder will take approx 2 minutes to display, during this time SMART FTP consumes a lot of CPU.

We have tested previous versions, and this issue was introduced after version 4.0.1072.0.
Rolling back to version 4.0.1072.0 and then opening the same directory takes appox 3 seconds to retrieve and display with very little resource.

We are using Lenovo Laptops all running Windows XP.
We can replicate on our T61, T400, and T410 model

#5 Stevezau

Stevezau
  • Members
  • 13 posts

Posted 01 June 2011 - 12:09 AM

Sorry for the late reply, was on holidays.

I have a few folders that contain around 4000 folders in each (not files).

The listing is taking ages.

Using lenovo w520 laptop.

Anyway i can do some troubleshooting??

PS: I should add that my default sorting is by the date column.

Edited by Stevezau, 01 June 2011 - 12:09 AM.


#6 mb

mb

    Developer

  • Administrators
  • 11521 posts

Posted 01 June 2011 - 12:10 AM

>Jess Peeble
On Pre-Windows Vista operating system we are using several workarounds to emulate features that are only available in newer operating systems. This causes the performance hit on these legacy operating systems (namely Windows XP and Windows 2003). One way to improve the performance is to install Windows Search on Windows XP / Windows 2003 from here: http://www.microsoft...&displaylang=en

On Windows Vista / Windows 2008 or higher there are no known issues and we are unable to reproduce the problem the starter of this thread has reported.

#7 mb

mb

    Developer

  • Administrators
  • 11521 posts

Posted 01 June 2011 - 12:11 AM

>Stevezau
Do you also see the problem with folders containing a lot of files but few folders?

#8 Stevezau

Stevezau
  • Members
  • 13 posts

Posted 01 June 2011 - 12:31 AM

mb,

I created a new folder with 7000 files and it was very quick..

I then created another folder (for testing purposes) with 7000 sub folders and it is taking much longer to sort but strangely enough my other folders which have 3000-4000 sub folders (each sub folder has a different modification date and size) are taking longer?

#9 Stevezau

Stevezau
  • Members
  • 13 posts

Posted 01 June 2011 - 12:32 AM

I've just noticed that it appears right after it downloads the listing the folders will appear and I can scroll for 1-2 seconds then smartftp freezes up... Maybe it's sorting them by date in the background or something?

#10 mb

mb

    Developer

  • Administrators
  • 11521 posts

Posted 01 June 2011 - 12:55 AM

>Stevezau
Correct. Sorting is done in the background. I will try to reproduce it with 100k folders (instead of files). Thanks for testing.

#11 mb

mb

    Developer

  • Administrators
  • 11521 posts

Posted 01 June 2011 - 02:03 AM

>Stevezau
I was able to reproduce it now. The problem is not the sorting itself (it is fast) but the creation of the internal cache/tree structure.

#12 Stevezau

Stevezau
  • Members
  • 13 posts

Posted 01 June 2011 - 02:04 AM

Ah ok.

Is this something that can be optimized??

#13 mb

mb

    Developer

  • Administrators
  • 11521 posts

Posted 01 June 2011 - 02:23 AM

Yes there is a lot of room for performance improvements.

#14 Stevezau

Stevezau
  • Members
  • 13 posts

Posted 01 June 2011 - 02:24 AM

Ok, is there somewhere i can log the bug?

#15 mb

mb

    Developer

  • Administrators
  • 11521 posts

Posted 01 June 2011 - 03:45 AM

It's not really necessary since we are aware of the bug now. But if you wish to do for completeness: http://www.smartftp....x.php?/tracker/

#16 mb

mb

    Developer

  • Administrators
  • 11521 posts

Posted 01 June 2011 - 07:49 AM

I made some improvements (available in the next build) in the tree control with the following results:

10'000 folders:
before : total: 37 sec,(8 sec read+list, 29 sec tree)
after : total: 10 sec (5 sec read+list, 5 sec tree)

#17 mb

mb

    Developer

  • Administrators
  • 11521 posts

Posted 01 June 2011 - 08:48 PM

The new version is available now:
http://www.smartftp.com/download/

#18 Jess Peebles

Jess Peebles
  • Members
  • 2 posts

Posted 01 June 2011 - 09:02 PM

The new version is available now:
http://www.smartftp.com/download/


Hi,
This new build 4.0.1186.0 has resolved my issues, Thanks!

#19 Stevezau

Stevezau
  • Members
  • 13 posts

Posted 01 June 2011 - 11:50 PM

New version fixes the issue for me as well..

what took 2-3mins now takes 5-6 seconds.

Thanks for the fast response!

Edited by Stevezau, 01 June 2011 - 11:50 PM.