I have never experienced this, in fact, quite the opposite.  Varnish does not require much memory to operate efficiently, I am not sure in what ways you have been using it.  Also, I would still use Varnish in front of an nginx setup, varnish does way better as a cache and has a lot more options...