Even when an identity file (i.e. a private key) is specified, SSH
tries all keys loaded by the SSH agent first for authentication
(if an agent is running).
When the agent returns many keys this can lead to a "Too Many Failures"
auth error, because none of the agent keys are accepted by the VM SSH
server.
Fix this by only using the provided key.