Multiple Proxy Syndrome

HalfEatenPie

The Irrational One
Retired Staff
So...  This may or may not look a bit sketchy, I totally understand (and it's totally reasonable to think of it as such).  

But my network right now (in Asia) is absolutely annoying internationally.  Recently (thanks to @TheLinuxBug) I've found some pretty interesting things about my own networking and I'd like to setup a proxy server that has the beginning point in one of my servers and the end-point as the last server.

E.g.

Me -> Japan -> LA -> Endpoint

What's the easiest way to do this?  My assumption was to setup multiple squid proxies or something But...  Yeah.... 

Any ideas?  
 

KuJoe

Well-Known Member
Verified Provider
Why not run a VPN client/server on each VPS and have them connect to each other?

VPN Client (You) -> VPN Server (Japan) -> VPN Client (Japan) -> VPN Server (LA) -> VPN Client (LA) -> VPN Server (Endpoint)
 

splitice

Just a little bit crazy...
Verified Provider
Server to server wise just use a GRE/IPIP tunnel, far better than the overhead from multiple VPN's....
 

HalfEatenPie

The Irrational One
Retired Staff
Why not run a VPN client/server on each VPS and have them connect to each other?

VPN Client (You) -> VPN Server (Japan) -> VPN Client (Japan) -> VPN Server (LA) -> VPN Client (LA) -> VPN Server (Endpoint)
The thing is that this isn't for 100% of the traffic.  Therefore it's more like selective routing for certain DNS and services rather than that.  Therefore I'd prefer to use a proxy or something.

However...  That wouldn't be too bad of an idea between the two servers. 

Server to server wise just use a GRE/IPIP tunnel, far better than the overhead from multiple VPN's....
Hm... that would be a possibility.  Granted I'd have to find more detailed literature on it since I haven't actually used it yet in application.  Out of curiosity, do you have any resources available that could guide me in the right direction?  
 

splitice

Just a little bit crazy...
Verified Provider
Ive never tried it. but just a plain old gre tunnel and when you setup the default route for the vpn just use the gre* interface instead of eth0.

You will also need to reduce the MTU for each hop (you will need to do that no matter which method chosen).
 
Last edited by a moderator:

Hxxx

Active Member
Interested in this:

Recently (thanks to @TheLinuxBug) I've found some pretty interesting things about my own networking 
 

HalfEatenPie

The Irrational One
Retired Staff
Interested in this:

Recently (thanks to @TheLinuxBug) I've found some pretty interesting things about my own networking 
Ahh.  Well it was simply a TCP Proxy system.  Connection to one port is redirected to a connect to another port on another server (that general thing).  Nothing new really but helped me out a ton when I was transferring 200 GB worth of data that originally would have taken a few days to a few hours by setting my own "route".  @TheLinuxBug definitely knows his stuff haha.  

Yeah I should look into that as a possibility.  
 

TheLinuxBug

New Member
I think there is a little bit of confusion about what I had assisted @HalfEatenPie with and what this thread is about:

Let me clarify a bit.  He was trying to move a large amount of data from one side of the US to another.  The source and destination did not share very good routes.  As such, I offered a middle point that had good peering to both ends which could provide full throughput from one server to the mid-point and the mid-point could provide full speed to the end point (100Mbit).

There are a lot of uses for VPS servers, but one I think that is often overlooked (at least in the way I have utilized some of my servers) is using the different locations and their peering to route data most efficiently from one point to another.  Now, I have used VPN for this, but in most cases you can utilize programs like (and here I am giving away some trade secrets, thanks @HalfEatenPie :huh:) 'redir', 'lamefwd' and even iptables to simply redirect local ports on a server to a new destination.  Doing this, will allow you to leverage the link of that server to send data on to the end point.  

When I buy a new server I take specific note to the peering and what providers are in use.  This allows me to utilize different servers which peer better to one another to pass data along.  In the case of @HalfEatenPie he had a server in North Carolina and another in Los Angeles, CA.  The server in LA did not share a particularly fast route to the server in NC, as such his transfers kept bursting and then dropping off to next to nothing (likely an oversold route which just couldn't sustain the high speed).  To help facilitate better speeds for the transfers, I offered up a server I had some available bandwidth on in Phoenix Nap which happened to peer well to both locations.  As such, @HalfEatenPie was able to realize the top speed of the servers in both locations and make the transfer at almost full 100Mbit.

One things to note when doing something like this is that you need to realize when you use a server like this, especially if they track bandwidth in both directions, if you transfer 100Gb through the server, you will actually be using 200GB as you will be charged for the data both incoming to the server and outgoing.  In this case I had the data to spare, but if you are not careful to consider this, you can quickly run out of bandwidth.

Now, the real way to do this in a real time environment would be to setup an VPN or GRE tunnel as mentioned by a few people here and then use something like Zebra to setup your own BGP routing table.  If you spend the time to setup something like this across a bunch of servers, you can actually start to create your own best routes and route your own data across your VPN network.  This method obviously takes a lot more of a time investment as you need to learn BGP routing and how to utilize a vpn network to route data, but in the end you can create your own routes across your network of servers and truly realize the benefits of being able to make your own routes.  Learning this will also help you if you wish to setup your own IPv6 routing or if you want to setup your own form of a 'darknet'.

So ends today's lesson on port redirection tools, vpns and BGP routing.  B)

Cheers!
 
Last edited by a moderator:

HalfEatenPie

The Irrational One
Retired Staff
I think there is a little bit of confusion about what I had assisted @HalfEatenPie with and what this thread is about:

Let me clarify a bit.  He was trying to move a large amount of data from one side of the US to another.  The source and destination did not share very good routes.  As such, I offered a middle point that had good peering to both ends which could provide full throughput from one server to the mid-point and the mid-point could provide full speed to the end point (100Mbit).

There are a lot of uses for VPS servers, but one I think that is often overlooked (at least in the way I have utilized some of my servers) is using the different locations and their peering to route data most efficiently from one point to another.  Now, I have used VPN for this, but in most cases you can utilize programs like (and here I am giving away some trade secrets, thanks @HalfEatenPie :huh:) 'redir', 'lamefwd' and even iptables to simply redirect local ports on a server to a new destination.  Doing this, will allow you to leverage the link of that server to send data on to the end point.  

When I buy a new server I take specific note to the peering and what providers are in use.  This allows me to utilize different servers which peer better to one another to pass data along.  In the case of @HalfEatenPie he had a server in North Carolina and another in Los Angeles, CA.  The server in LA did not share a particularly fast route to the server in NC, as such his transfers kept bursting and then dropping off to next to nothing (likely an oversold route which just couldn't sustain the high speed).  To help facilitate better speeds for the transfers, I offered up a server I had some available bandwidth on in Phoenix Nap which happened to peer well to both locations.  As such, @HalfEatenPie was able to realize the top speed of the servers in both locations and make the transfer at almost full 100Mbit.

One things to note when doing something like this is that you need to realize when you use a server like this, especially if they track bandwidth in both directions, if you transfer 100Gb through the server, you will actually be using 200GB as you will be charged for the data both incoming to the server and outgoing.  In this case I had the data to spare, but if you are not careful to consider this, you can quickly run out of bandwidth.

Now, the real way to do this in a real time environment would be to setup an VPN or GRE tunnel as mentioned by a few people here and then use something like Zebra to setup your own BGP routing table.  If you spend the time to setup something like this across a bunch of servers, you can actually start to create your own best routes and route your own data across your VPN network.  This method obviously takes a lot more of a time investment as you need to learn BGP routing and how to utilize a vpn network to route data, but in the end you can create your own routes across your network of servers and truly realize the benefits of being able to make your own routes.  Learning this will also help you if you wish to setup your own IPv6 routing or if you want to setup your own form of a 'darknet'.

So ends today's lesson on port redirection tools, vpns and BGP routing.  B)

Cheers!
And this is why I love you in a totally manly way.  

haha my bad with giving away your secret.  I owe you
 
Last edited by a moderator:
Top