File list omits some files - limited to 1000?

raylutz

2008-02-18 22:14:54

Running on XP, files on remote are not all listed.

Here is file list using PUTTY:

[copswiki@rpdz2 Members]$ ls P[89]*.txt
P800.txt P818.txt P835.txt P852.txt P86.txt P888.txt P904.txt P921.txt P939.txt P956.txt P973.txt P990.txt
P801.txt P819.txt P836.txt P853.txt P870.txt P889.txt P905.txt P922.txt P93.txt P957.txt P974.txt P991.txt
P802.txt P81.txt P837.txt P854.txt P871.txt P88.txt P906.txt P923.txt P940.txt P958.txt P975.txt P992.txt
P803.txt P820.txt P838.txt P855.txt P872.txt P890.txt P907.txt P924.txt P941.txt P959.txt P976.txt P993.txt
P804.txt P821.txt P839.txt P856.txt P873.txt P891.txt P908.txt P925.txt P942.txt P95.txt P977.txt P994.txt
P805.txt P822.txt P83.txt P857.txt P874.txt P892.txt P909.txt P926.txt P943.txt P960.txt P978.txt P995.txt
P806.txt P823.txt P840.txt P858.txt P875.txt P893.txt P90.txt P927.txt P944.txt P961.txt P979.txt P996.txt
P807.txt P824.txt P841.txt P859.txt P876.txt P894.txt P910.txt P928.txt P945.txt P962.txt P97.txt P997.txt
P808.txt P825.txt P842.txt P85.txt P877.txt P895.txt P911.txt P929.txt P946.txt P963.txt P980.txt P998.txt
P809.txt P826.txt P843.txt P860.txt P878.txt P896.txt P912.txt P92.txt P947.txt P964.txt P981.txt P999.txt
P80.txt P827.txt P844.txt P861.txt P879.txt P897.txt P913.txt P930.txt P948.txt P965.txt P982.txt P99.txt
P810.txt P828.txt P845.txt P862.txt P87.txt P898.txt P914.txt P931.txt P949.txt P966.txt P983.txt P9.txt
P811.txt P829.txt P846.txt P863.txt P880.txt P899.txt P915.txt P932.txt P94.txt P967.txt P984.txt
P812.txt P82.txt P847.txt P864.txt P881.txt P89.txt P916.txt P933.txt P950.txt P968.txt P985.txt
P813.txt P830.txt P848.txt P865.txt P882.txt P8.txt P917.txt P934.txt P951.txt P969.txt P986.txt
P814.txt P831.txt P849.txt P866.txt P883.txt P900.txt P918.txt P935.txt P952.txt P96.txt P987.txt
P815.txt P832.txt P84.txt P867.txt P884.txt P901.txt P919.txt P936.txt P953.txt P970.txt P988.txt
P816.txt P833.txt P850.txt P868.txt P885.txt P902.txt P91.txt P937.txt P954.txt P971.txt P989.txt
P817.txt P834.txt P851.txt P869.txt P886.txt P903.txt P920.txt P938.txt P955.txt P972.txt P98.txt

Note files starting with P8 and P9 are shown. But in raw listing (and in regular display) of SmartFTP files are omitted:

Here is raw list...
type=file;size=1458;modify=20071014220455;UNIX.mode=0644;UNIX.uid=32093;UNIX.gid
=32095;unique=807g2bf0045; P0.txt
type=file;size=2011;modify=20071014220456;UNIX.mode=0644;UNIX.uid=32093;UNIX.gid
=32095;unique=807g2bf0046; P1.txt
type=file;size=3707;modify=20071014220456;UNIX.mode=0444;UNIX.uid=32093;UNIX.gid
=32095;unique=807g2bf0047; P1.txt,v
type=file;size=2094;modify=20071014220513;UNIX.mode=0644;UNIX.uid=32093;UNIX.gid
=32095;unique=807g2bf0057; P10.txt
type=file;size=5231;modify=20071014220515;UNIX.mode=0444;UNIX.uid=32093;UNIX.gid
=32095;unique=807g2bf0058; P10.txt,v
type=file;size=2660;modify=20080121035801;UNIX.mode=0644;UNIX.uid=32093;UNIX.gid
=32095;unique=807g2bf0100; P100.txt
type=file;size=6996;modify=20080121035801;UNIX.mode=0444;UNIX.uid=32093;UNIX.gid
=32095;unique=807g2bf0101; P100.txt,v
type=file;size=2307;modify=20080126205318;UNIX.mode=0644;UNIX.uid=32093;UNIX.gid
=32095;unique=807g2bf1138; P1000.txt
type=file;size=2497;modify=20080126205318;UNIX.mode=0444;UNIX.uid=32093;UNIX.gid
=32095;unique=807g2bf1139; P1000.txt,v
type=file;size=2305;modify=20080126205417;UNIX.mode=0644;UNIX.uid=32093;UNIX.gid
=32095;unique=807g2bf113a; P1001.txt
type=file;size=2495;modify=20080126205417;UNIX.mode=0444;UNIX.uid=32093;UNIX.gid
=32095;unique=807g2bf113b; P1001.txt,v
type=file;size=2298;modify=20080126205507;UNIX.mode=0644;UNIX.uid=32093;UNIX.gid
=32095;unique=807g2bf113c; P1002.txt
...snip...
type=file;size=2339;modify=20080210224454;UNIX.mode=0644;UNIX.uid=32093;UNIX.gid
=32095;unique=807g2bf122b; P1138.txt
type=file;size=2528;modify=20080210224454;UNIX.mode=0444;UNIX.uid=32093;UNIX.gid
=32095;unique=807g2bf122c; P1138.txt,v
type=file;size=2333;modify=20080210230821;UNIX.mode=0644;UNIX.uid=32093;UNIX.gid
=32095;unique=807g2bf121a; P1139.txt
type=file;size=2522;modify=20080210230821;UNIX.mode=0444;UNIX.uid=32093;UNIX.gid
=32095;unique=807g2bf122d; P1139.txt,v
type=file;size=2533;modify=20080211210550;UNIX.mode=0644;UNIX.uid=32093;UNIX.gid
=32095;unique=807g2bf0118; P114.txt
type=file;size=6247;modify=20080211210550;UNIX.mode=0444;UNIX.uid=32093;UNIX.gid
=32095;unique=807g2bf0119; P114.txt,v
type=file;size=1992;modify=20071014220828;UNIX.mode=0644;UNIX.uid=32093;UNIX.gid
=32095;unique=807g2bf011a; P115.txt
type=file;size=3669;modify=20071014220829;UNIX.mode=0444;UNIX.uid=32093;UNIX.gid
=32095;unique=807g2bf011b; P115.txt,v
type=file;size=2472;modify=20080218205247;UNIX.mode=0644;UNIX.uid=32093;UNIX.gid
=32095;unique=807g2bf011c; P116.txt
type=file;size=5359;modify=20080218205247;UNIX.mode=0444;UNIX.uid=32093;UNIX.gid
=32095;unique=807g2bf011d; P116.txt,v
type=file;size=1486;modify=20071014220832;UNIX.mode=0644;UNIX.uid=32093;UNIX.gid
=32095;unique=807g2bf011e; P117.txt
type=file;size=1986;modify=20071014220833;UNIX.mode=0644;UNIX.uid=32093;UNIX.gid
=32095;unique=807g2bf011f; P118.txt
type=file;size=3655;modify=20071014220834;UNIX.mode=0444;UNIX.uid=32093;UNIX.gid
=32095;unique=807g2bf0120; P118.txt,v
type=file;size=2023;modify=20071014220834;UNIX.mode=0644;UNIX.uid=32093;UNIX.gid
=32095;unique=807g2bf0121; P119.txt
type=file;size=2522;modify=20071014220835;UNIX.mode=0444;UNIX.uid=32093;UNIX.gid
=32095;unique=807g2bf0122; P119.txt,v
type=file;size=1711;modify=20071014220518;UNIX.mode=0644;UNIX.uid=32093;UNIX.gid
=32095;unique=807g2bf005b; P12.txt
type=file;size=3096;modify=20071014220519;UNIX.mode=0444;UNIX.uid=32093;UNIX.gid
=32095;unique=807g2bf005c; P12.txt,v
type=file;size=1983;modify=20071014220836;UNIX.mode=0644;UNIX.uid=32093;UNIX.gid
=32095;unique=807g2bf0123; P120.txt
type=file;size=3943;modify=20071014220837;UNIX.mode=0444;UNIX.uid=32093;UNIX.gid
=32095;unique=807g2bf0124; P120.txt,v
type=file;size=1973;modify=20071014220838;UNIX.mode=0644;UNIX.uid=32093;UNIX.gid
=32095;unique=807g2bf0125; P121.txt
type=file;size=3629;modify=20071014220839;UNIX.mode=0444;UNIX.uid=32093;UNIX.gid
=32095;unique=807g2bf0126; P121.txt,v
type=file;size=1996;modify=20071014220841;UNIX.mode=0644;UNIX.uid=32093;UNIX.gid
=32095;unique=807g2bf0127; P122.txt
...snip...
type=file;size=2093;modify=20071029202311;UNIX.mode=0644;UNIX.uid=32093;UNIX.gid
=32095;unique=807g2bf0fa1; P853.txt
type=file;size=2283;modify=20071029202311;UNIX.mode=0444;UNIX.uid=32093;UNIX.gid
=32095;unique=807g2bf0fa2; P853.txt,v
type=file;size=2109;modify=20071029202600;UNIX.mode=0644;UNIX.uid=32093;UNIX.gid
=32095;unique=807g2bf0fa3; P854.txt
type=file;size=2299;modify=20071029202600;UNIX.mode=0444;UNIX.uid=32093;UNIX.gid
=32095;unique=807g2bf0fa4; P854.txt,v
type=file;size=2121;modify=20071029202653;UNIX.mode=0644;UNIX.uid=32093;UNIX.gid
=32095;unique=807g2bf0fa5; P855.txt
type=file;size=2311;modify=20071029202653;UNIX.mode=0444;UNIX.uid=32093;UNIX.gid
=32095;unique=807g2bf0fa6; P855.txt,v
(THIS IS THE END)

You can see that files P856.txt through P999.txt exist in the file listing but NOT in the SmartFTP listing. I have not changed any listing filters (plus it's a pretty strange thing to miss.)

However, the files from P856 to P999 would be the last ones in a sorted list, and so it is my hunch that SmartFTP is limited to perhaps 1000 files in a listing.

I tested this theory by adding another file. It took the first unused number, P887. But now the listing is truncated at P855.txt. Test confirms that theory is probably correct.

ALSO... files involved in a transfer are also limited to this set. IT DOES NOT TRANSFER ANY OVER THE 1000 SHOWN, even though they are in the folder!!

Advice?

--Raymond

P.S. THANKS For a great product!!

2008-02-18 23:16:02

Files that are not returned in the directory listing by the server cannot be displayed by SmartFTP. Putty uses LIST and no MLSD that's why you have a different situation. To force SmartFTP to use the LIST command instead of the newer/better MLSD command go to the favorite settings. Then in the FTP->Transfer dialog disable the [x] Use MLSD option. This will workaround the bug with the FTP server not returning the complete or incorrect MLSD listing.
Please post the log if this change shouldn't work.

Regards,
Mat

raylutz

2008-02-19 23:38:33

Files that are not returned in the directory listing by the server cannot be displayed by SmartFTP. Putty uses LIST and no MLSD that's why you have a different situation. To force SmartFTP to use the LIST command instead of the newer/better MLSD command go to the favorite settings. Then in the FTP->Transfer dialog disable the [x] Use MLSD option. This will workaround the bug with the FTP server not returning the complete or incorrect MLSD listing.
Please post the log if this change shouldn't work.

Regards,
Mat

I changed the setting to MLSD "disabled". Results are the same sort of response. I notice this in the transaction log:
[15:22:57] LIST -aL
[15:22:57] 150 Accepted data connection
[15:22:58] 226-Options: -a -l
[15:22:58] 226 Output truncated to 2000 matches

I set it back to MLSD:

[15:26:56] MLSD
[15:26:56] 150 Accepted data connection
[15:26:57] 226-Options: -a -l
[15:26:57] 226 Output truncated to 2000 matches
[15:26:57] 236617 bytes transferred. (214 KB/s) (00:00:01)

It might be good to warn the user of SmartFTP that the results are incomplete when they are truncated. I will contact my hosting service to see if FTP settings can be modified. Is there any work around (other than archive file or zip file) if there are >2000 files in the directory? (That is correct because each one of the files also has a RCS version .txt,v. The ls command I used only listed the .txt files).

2008-02-20 00:09:34

There is no workaround on the client side because the server does not return more than 2000 files in the listing. I would recommend you to move the files from the big directory into sub directories.

Regards,
Mat