In order to have gnupg cache your passwords from gpg-agent, you must set ~/.gnupg/gpg.conf use-agent and in ~/.gnupg/gpg-agent.conf default-cache-ttl 3600 (min) max-cache-ttl 3600 (min) which is the time the password will be kept in the cache, and the maximum time it can be kept.