Sometimes you may do something to mess up these permissions, like transferring your keys to another machine – as I did. Run the following to fix most permissions problems. You may have to do this on both the remote host and local host.
chmod 600 ~/.ssh/id_rsa
chmod 644 ~/.ssh/id_rsa.pub