ElTopo
Hi,
Recently I need to transfer several huge files (10GB+ bytes), when SmartFTP client asks FTP server to return XMD5/XCRC checksums it takes very long time (sometimes over 20 minutes) for server to complete calculating the result and client gets timed out. I know I can turn off integrity checking for huge files (and I don't want to), or set longer timeout value, but I think it would be convenient if SmartFTP can check integrity in smaller blocks.
Let's say, for a 700MB file, user set a (configurable) block size to 128MB. When SmartFTP client needs to check integrity, it issues several commands:
XMD5 file 0 128MB-1
(then verify local file's MD5 for [0, 128MB-1])
XMD5 file 128MB, 256MB-1
(then verify local file's MD5 for [128MB, 256MB-1])
...
XMD5 file 640MB 700MB-1
(then verify local file's MD5 for [640MB, 700MB-1])
so instead of checking one 700MB file, now it checks 6 smaller blocks, saving the trouble of "timed out".
There's also a bonus for partial downloaded file: SmartFTP client can resume bad downloading from the first bad block. If I have a corrupted local 2.5GB file from the server side's 4GB file, SmartFTP client checks XMD5 in blocks, finds out that the first 2048MB part is good, it simply resume downloading from 2048MB+1 bytes.
Recently I need to transfer several huge files (10GB+ bytes), when SmartFTP client asks FTP server to return XMD5/XCRC checksums it takes very long time (sometimes over 20 minutes) for server to complete calculating the result and client gets timed out. I know I can turn off integrity checking for huge files (and I don't want to), or set longer timeout value, but I think it would be convenient if SmartFTP can check integrity in smaller blocks.
Let's say, for a 700MB file, user set a (configurable) block size to 128MB. When SmartFTP client needs to check integrity, it issues several commands:
XMD5 file 0 128MB-1
(then verify local file's MD5 for [0, 128MB-1])
XMD5 file 128MB, 256MB-1
(then verify local file's MD5 for [128MB, 256MB-1])
...
XMD5 file 640MB 700MB-1
(then verify local file's MD5 for [640MB, 700MB-1])
so instead of checking one 700MB file, now it checks 6 smaller blocks, saving the trouble of "timed out".
There's also a bonus for partial downloaded file: SmartFTP client can resume bad downloading from the first bad block. If I have a corrupted local 2.5GB file from the server side's 4GB file, SmartFTP client checks XMD5 in blocks, finds out that the first 2048MB part is good, it simply resume downloading from 2048MB+1 bytes.