Google Authenticator can be used with a VPS. I use it on a few of mine.
It doesn't use your Google account for login. Like walesmd said above, it's a pseudo random number generated from a unique hash.
Google provide a PAM module that can be quite easily implemented. There's a good howto...