I know this is a really old post, but I had a similar problem. I installed VestaCP without the DNS server and even if I set the "DKIM Support" option, the public key wasn't being generated, so I generated it my self with:
# /home/admin/conf/mail/your_domain.com
openssl rsa -in dkim.pem -out...