SFTP - Wrong Default CHMOD

When I connect to a server using SFTP it uploads all files with 664 permissions, which causes them to not work, but if I use normal FTP it uploads with 644 as the default permissions. How do I change the default permissions a file is uploaded with SFTP?

I think you have to change the umask on the server side.