I have never experienced this, in fact, quite the opposite. Varnish does not require much memory to operate efficiently, I am not sure in what ways you have been using it. Also, I would still use Varnish in front of an nginx setup, varnish does way better as a cache and has a lot more options...