amuck-landowner

$200 if somebody can solve this IP problem.

KuJoe

Well-Known Member
Verified Provider
So we've been having this problem for a while now but nobody has been able to assist us. We've hired experts, paid hundreds of dollars in remote hands for multiple data centers, and posted threads on various forums without any luck so I'm reaching out to vpsBoard in the hopes that somebody might have a solution to our problem.

The problem is that an IP that was once assigned to a network interface (be it physical or virtual) continues using traffic (normally over 50Mbps, sometimes over 100Mbps). The traffic is 100% symmetrical (i.e for every Bps outbound, there is an equal Bps inbound). When we reset the NIC the traffic stops for a few minutes and then returns. The IP does not ping and is not located in any tables on the server or the routers/switches. A tcpdump shows the traffic has little in common also, sometimes it'll be ICMP traffic and sometimes it'll be packets going to port 22 even though nothing is listening on port 22 on the server even if the IP was bound to a NIC.

I've changed every setting I can think of in sysctl and we've tried different servers, switches, routers, and NICs without any luck. The only common thing is that we always run CentOS 6.x as the OS (although we've tried multiple different kernels over the months we've been working on this problem). I've used iptables to drop all packets to the IPs (source and destination) but the traffic still continues. I've even gone as far to reboot the server but the traffic returns after the server comes back online. We thought this problem was limited to OpenVZ but we had an IP bound to eth0:0 and after I removed it the traffic jumped to 140Mbps. I tested it on a KVM VPS (installed OpenVZ and set it up just like our other OpenVZ nodes) with no luck either. We used to think the problem was ARP related but this is not the case since flushing the ARP tables on the server or network devices have no effect.

I am willing to pay $200 via Paypal if somebody can explain it and provide a solution that does not involve purchasing additional equipment.
 

Mun

Never Forget
A thing that comes to mind is that there is another server inside your network with that same IP that you don't know about. Like a KVM VPS that has taken over another IP. You might be able to do this with Bird *if I remember correctly* and use a similar setup to cloudflare.

I could be very wrong but that is all I can think of from the info you provided. 
 

KuJoe

Well-Known Member
Verified Provider
I can guarantee the IP isn't in use anywhere else on the network (we had our data centers check their routers to confirm).
 

KuJoe

Well-Known Member
Verified Provider
IPs are always different. Sometimes a single IP that is offline will have dozens of external IPs connecting to it from different countries.
 

Hxxx

Active Member
ok I don't know if I read this months ago from you. But I think somebody solved this over WHT. Can't recall. Maybe it was you doing this question.
 

KuJoe

Well-Known Member
Verified Provider
I posted this on WHT a while back and the suggestions were either not possible for our networks or did not work.
 

vedalken254

New Member
IPs are always different. Sometimes a single IP that is offline will have dozens of external IPs connecting to it from different countries.
I'm going to fire up my trusty CentOS 6 server. Is there any chance you could either post or pm me a link to a traffic dump? Not knowing a lot about the situation makes it somewhat hard to accurately diagnose. Also, was this IP (the one supposedly attached to your server) at one point attached to a VM or anything on that Server?
 

Francisco

Company Lube
Verified Provider
Give me some sort of TCPDUMP access and a copy of the switch config in PM.

If I figure it out you can just buy Manndude a pizza and we'll call it even :p

Francisco
 
Last edited by a moderator:

vedalken254

New Member
I would assume (based on your OP) that the ip does belong to you (the one receiving and sending traffic) and by your statement, I assume you checked your net.ipv4.ip_forward setting as well as trying brctl show? I don't know if you host VPS' still but the main thing I can think of is a rogue bridge or something of that nature.
 

KuJoe

Well-Known Member
Verified Provider
I'm going to fire up my trusty CentOS 6 server. Is there any chance you could either post or pm me a link to a traffic dump? Not knowing a lot about the situation makes it somewhat hard to accurately diagnose. Also, was this IP (the one supposedly attached to your server) at one point attached to a VM or anything on that Server?
Here's a snippet from the tcpdump from tonight, I didn't grab it so I'm not sure if we have any more than this:


# tcpdump -n dst 162.211.66.3
21:59:23.635018 IP 116.10.191.183.x11 > 162.211.66.3.ssh: Flags , seq 2075852800, win 16384, length 0
21:59:23.635105 IP 116.10.191.183.x11 > 162.211.66.3.ssh: Flags , seq 2075852800, win 16384, length 0
21:59:23.635122 IP 116.10.191.183.x11 > 162.211.66.3.ssh: Flags , seq 2075852800, win 16384, length 0
21:59:23.635215 IP 116.10.191.183.x11 > 162.211.66.3.ssh: Flags , seq 2075852800, win 16384, length 0
21:59:23.635233 IP 116.10.191.183.x11 > 162.211.66.3.ssh: Flags , seq 2075852800, win 16384, length 0
21:59:23.635319 IP 116.10.191.183.x11 > 162.211.66.3.ssh: Flags , seq 2075852800, win 16384, length 0
21:59:23.635335 IP 116.10.191.183.x11 > 162.211.66.3.ssh: Flags , seq 2075852800, win 16384, length 0
21:59:23.635427 IP 116.10.191.183.x11 > 162.211.66.3.ssh: Flags , seq 2075852800, win 16384, length 0

The IP in question here (162.211.66.3) was assigned to eth0:0 on the node, I removed eth0:0 and the flood of traffic started.

Give me some sort of TCPDUMP access and a copy of the switch config in PM.


If I figure it out you can just buy Manndude a pizza and we'll call it even :p


Francisco
We don't have access to the switch configs (we don't run our own switches outside of Tampa). I'll try to get a better tcpdump next time this occurs.

I would assume (based on your OP) that the ip does belong to you (the one receiving and sending traffic) and by your statement, I assume you checked your net.ipv4.ip_forward setting as well as trying brctl show? I don't know if you host VPS' still but the main thing I can think of is a rogue bridge or something of that nature.
Yes, the IP does belong to us and net.ipv4.ip_forward is set to one as required by OpenVZ to work. We do not have any bridges so brctl show returns an empty table.
 

KuJoe

Well-Known Member
Verified Provider
Additionally, when we add the IP to an interface or a VPS the traffic stops immediately.

Also, netstat -plan does not show any traffic for the destination or the source IP when this occurs. Any ideas why we can view the traffic in iftop but not netstat?
 
Last edited by a moderator:

vedalken254

New Member
Thanks for the info. I think I have an answer as for why, but not how to fix it just yet. The issue (I THINK) lies with the IP being disconnected from an interface and that because the server is somehow still receiving the packets, the data in/out is symmetrical because iptables is like, "this ip isn't on my ifconfig -a output so why the heck am I getting data for it?!" thus sending (most likely an ICMP response to the offending IP. I've had a similar occurrence when downloading linux distros via BitTorrent. If I close the client without cutting the network connection (aka the download) on the torrent itself, i get SYN flooded according to most Winderps Firewalls that have Network Attack Blocker functionality (Kaspersky being the major one). The problem in your case is the flood isn't stopping. Also, adding the IP to an interface or a VPS stopping the traffic flow indicates a faulty route somewhere. Have you tried assigning it to a VPS and then running iftop on the VPS also?
 
Last edited by a moderator:

KuJoe

Well-Known Member
Verified Provider
I'll try running iftop on the new VPS after the IP is added and see if I see anything there.
 

vedalken254

New Member
I'll try running iftop on the new VPS after the IP is added and see if I see anything there.
Okay. Also, the issue i've found out about that .x11 ip is that 1) it's originating from China and 2) several sites pin it as an SSH bruteforcer IP. As for why you're continuing to receive traffic on the host node even though the IP isn't assigned to it, i don't know as of yet.
 

rds100

New Member
Verified Provider
Here's a snippet from the tcpdump from tonight, I didn't grab it so I'm not sure if we have any more than this:


# tcpdump -n dst 162.211.66.3
21:59:23.635018 IP 116.10.191.183.x11 > 162.211.66.3.ssh: Flags , seq 2075852800, win 16384, length 0
21:59:23.635105 IP 116.10.191.183.x11 > 162.211.66.3.ssh: Flags , seq 2075852800, win 16384, length 0
21:59:23.635122 IP 116.10.191.183.x11 > 162.211.66.3.ssh: Flags , seq 2075852800, win 16384, length 0
21:59:23.635215 IP 116.10.191.183.x11 > 162.211.66.3.ssh: Flags , seq 2075852800, win 16384, length 0
21:59:23.635233 IP 116.10.191.183.x11 > 162.211.66.3.ssh: Flags , seq 2075852800, win 16384, length 0
21:59:23.635319 IP 116.10.191.183.x11 > 162.211.66.3.ssh: Flags , seq 2075852800, win 16384, length 0
21:59:23.635335 IP 116.10.191.183.x11 > 162.211.66.3.ssh: Flags , seq 2075852800, win 16384, length 0
21:59:23.635427 IP 116.10.191.183.x11 > 162.211.66.3.ssh: Flags , seq 2075852800, win 16384, length 0



This seems like the same packet cycling until the TTL runs out.

Try tcpdump -v -v -v to see if these packets differ by the ttl (decreasing with each next).

What happens according to me:

- Rogue random packet comes from the internet (port scans, whatever)

- Your router, L3 switch or whatever is the gateway device to your network has a cached ARP packet for this IP and is sending the packet to the MAC address of your server.

- Your server is receiving the packet (because it's destined to it's MAC address), but doesn't recognize the destination IP as it's own so has to forward the packet.

- The packet is forwarded according to the routing table (i.e. to the default gateway), and the TTL is decreased by 1

- The default gateway (your router / L3 switch / whatever) receives the packet and routes it back to your server (again decreasing the TTL by 1)

- repeat until the TTL expires (reaches zero).

Do you have control over the router (or whatever the gateway to your network is)? Can you check or clear the ARP cache there?
 

rds100

New Member
Verified Provider
And if indeed this is true (checked by the tcpdump -v -v -v option and seeing decreased TTL with each next copy of the packet) there are several ways to solve it that i can think of:

1. delete the ARP for 162.211.66.3 from your router.

2. iptables -I FORWARD -j DROP -d 162.211.66.3

3. modprobe dummy ; ifconfig dummy0 up ; ip route add 162.211.66.3 dev dummy0

#1 would be the best, but might be out of your control if you don't have your own router there.

with #2 and #3 you would still receive the packet, but only once. It would not cycle back and forth until the TTL expires.
 
Last edited by a moderator:

vedalken254

New Member
This seems like the same packet cycling until the TTL runs out.

Try tcpdump -v -v -v to see if these packets differ by the ttl (decreasing with each next).

What happens according to me:

- Rogue random packet comes from the internet (port scans, whatever)

- Your router, L3 switch or whatever is the gateway device to your network has a cached ARP packet for this IP and is sending the packet to the MAC address of your server.

- Your server is receiving the packet (because it's destined to it's MAC address), but doesn't recognize the destination IP as it's own so has to forward the packet.

- The packet is forwarded according to the routing table (i.e. to the default gateway), and the TTL is decreased by 1

- The default gateway (your router / L3 switch / whatever) receives the packet and routes it back to your server (again decreasing the TTL by 1)

- repeat until the TTL expires (reaches zero).

Do you have control over the router (or whatever the gateway to your network is)? Can you check or clear the ARP cache there?
I never even thought about that. Kudos. However, if he actually disabled the interface that the ip was associated with after removing the IP, wouldn't the result not be a TTL Loop?

Just a thought.

Veddy

EDIT: He's stated that he has no control over the router/L3 switch in a previous post. But that should be something the DC can fix for him.

2nd EDIT: OP states that he's had the ARP tables flushed so this probably isn't it now that i think about it. There's still the off-chance that it is, but i dunno.
 
Last edited by a moderator:

rds100

New Member
Verified Provider
He hasn't disabled eth0 (and wouldn't do it of course), and that's where the packet is coming from and going to. He has just removed some ip alias, i.e. removed the ip from the box. Which means now it's routed, instead of handled locally.

@KuJoe 162.211.66.0/24 (or whetever the maks) is probably not your main subnet, right? It's an additional subnet added to your VLAN and eth0 is not from this subnet?

You should do

ip route add 162.211.66.0/24 dev eth0

Again adjust the subnet mask accordingly.
 
Last edited by a moderator:
Top
amuck-landowner