That's pretty clever, though a transparent squid proxy is a bit more than I'd put on most of my vpn hosts due to ram requirements.
I think the problem becomes detecting torrenting; most, if not all, modern clients at least try to use encrypted connections, which you pretty much have to allow because SSL is used to secure a lot of applications. From the L7 filter page, matching for bittorrent is fickle at best, it won't positively ID all bt connections and time to classification is bad.
Honestly, I would handle this with 4 tools,
- QoS - known accepted traffic is prioritized above all others. A single user or group of users cannot be allowed to hose the entire service under any circumstance.
- Accountability - Keep traffic logs for quantity of data transferred, samplings of number of active connections from conntrack, when a high number of concurrents include a sample of connections list from conntrack, login ip connected from, login time, associated NAT or public ip if more than one available, logout time, and anything else you find appropriate. Require communication with clients be retained through a unified, searchable system including automated mailers, helpdesk tickets, and staff email, tagged by client.
- Education - Let your users know, be it through a once-per-login transparent squid redirect that shows a ToS/AUP, pre-notify them through signup, monthly newsletter, or what have you. Let them know that you're doing what you can to make the service as high performance as possible for legitimate uses. Be clear about what you think those uses are. Be clear about what you think is unacceptable. Let them know in your privacy policy what information you collect, why, for how long, and explicitly state how it is used and can be used by you. Keep your ToS/AUP up to date and notify users of changes including a brief summary of the change. Be succinct enough that users will at least briefly read it and verbose enough to cover the topic. Most users will listen; Ban the ones who don't.
- Respectability - Stick to your guns; if your TOS says zero tolerance, enforce zero tolerance, no leeway, no exceptions. Back up your privacy policy with action; don't give out member info without a judge-issued warrant. If you suspect a user is defrauding, botnetting, CPing, or otherwise committing felonious activity, gather data and report to the police/FBI (note that you have the ability, authority, and willingness to do this in your privacy policy). Never ignore suspicion as cause to investigate further but never act without hard, actionable evidence. Enforcing policy is the one place you should really be a hardass, but never one inch beyond that line. Keep your systems up to date and secure; don't ever lose your customers'/clients' trust because of data breach, and if it happens, you have to let them know so that they can take appropriate action to defend themselves. Bottom line is this: You are ultimately responsible for what happens on your systems including the activities of your users and your users for their accounts regardless of any incident or circumstance. Hold yourself to that standard and your users as well and all will be good.
This changes a technical solution into a social engineering solution. It takes significantly more maintenance effort to do this, as opposed to implementing a few firewall rules, but it's a lot more flexible for your users and provides a much higher quality service.