Can't Download (Transfer) Files on Some FTP's

I hope someone can help. I'm a little experienced with FTP programs but can't seem to find an answer to a problem I encounter now and then. I can log on fine to some sites but when I attempt to download a file, SmartFTP goes through its stages and gets as far as:

"227 Entering Passive Mode (32,6,358,907,56,81).
Opening data connection IP: 32,6,358,907,56,81 PORT: 4867.
RETR 123.iso
125 Data connection already open; Transfer starting."

but the transfer doesn't start. Nothing is shown to be transfering and eventually (after 30 seconds or so) the message "Connection closed. Server timeout." comes up. I've tried changing the mode as well as "Force Sever IP for Passive Mode" but no success. I've even tried different FTP programs but I encounter the same problem with them as well.
Does anybody have a solution?

(Note: As I mentioned before, this only happens with a minority of FTP sites.)