iptables interface forwarding, want to block bogons



my aim is to block so called "bogons" (http://iplists.firehol.org/?ipset=fullbogons)
I will do it thru ipset (probably with hash:net parameter as the total number of IPs is few hundred million)
But i need to block it on proper interface, because my server host few virtual servers whose main net interface is venet0

My dedicated OpenVZ VPS host server has these interfaces:
em1, lo, venet0

I could block it on all interfaces like this:
iptables -I INPUT -m set --match-set badips src -j DROP
iptables -I FORWARD -m set --match-set badips src -j DROP
(badips is an ipset set name)

But i have doubts not to block 127.**** or (vpn) kind of VPS traffic. On which interface/s i should block bogons on my dedicated host server please in my case?

Thank You