amuck-landowner

$200 if somebody can solve this IP problem.

V

vedalken254

New Member
rds100 said:
He hasn't disabled eth0 (and wouldn't do it of course), and that's where the packet is coming from and going to. He has just removed some ip alias, i.e. removed the ip from the box. Which means now it's routed, instead of handled locally.

@KuJoe 162.211.66.0/24 (or whetever the maks) is probably not your main subnet, right? It's an additional subnet added to your VLAN and eth0 is not from this subnet?

You should do

ip route add 162.211.66.0/24 dev eth0

Again adjust the subnet mask accordingly.
Click to expand...
No. Eth0 happens to be receiving the traffic I admit, however the IP was assigned to eth0:0 which typically means a virtual MAC address on the node. He claims he removed the eth0:0 interface and that's when this traffic flood started. If the VMAC is gone, why is the main physical MAC receiving the old VMAC's data? ARP caching wouldn't explain that.

EDIT: it seems that as of CentOS 6, they re-worked the ethx:y system. This is something I was unaware of. My apologies.

@KuJoe , Answer something for me: have you assigned the IP to a VPS on that node with the VPS having a different MAC address than the host (which should be always the case last I checked)? If so, this ARP cache crap wouldn't be the cause as the MAC address would be different and shouldn't go to the host node regardless.
 
Last edited by a moderator:
mikho

mikho

Not to be taken seriously, ever!
So the first question to the DC is to have them check in their arp table for the IP.


If it is there remove it.


If NOT, the problem is somewhere else.
 
V

vedalken254

New Member
KuJoe said:
So we've been having this problem for a while now but nobody has been able to assist us. We've hired experts, paid hundreds of dollars in remote hands for multiple data centers, and posted threads on various forums without any luck so I'm reaching out to vpsBoard in the hopes that somebody might have a solution to our problem.

The problem is that an IP that was once assigned to a network interface (be it physical or virtual) continues using traffic (normally over 50Mbps, sometimes over 100Mbps). The traffic is 100% symmetrical (i.e for every Bps outbound, there is an equal Bps inbound). When we reset the NIC the traffic stops for a few minutes and then returns. The IP does not ping and is not located in any tables on the server or the routers/switches. A tcpdump shows the traffic has little in common also, sometimes it'll be ICMP traffic and sometimes it'll be packets going to port 22 even though nothing is listening on port 22 on the server even if the IP was bound to a NIC.

I've changed every setting I can think of in sysctl and we've tried different servers, switches, routers, and NICs without any luck. The only common thing is that we always run CentOS 6.x as the OS (although we've tried multiple different kernels over the months we've been working on this problem). I've used iptables to drop all packets to the IPs (source and destination) but the traffic still continues. I've even gone as far to reboot the server but the traffic returns after the server comes back online. We thought this problem was limited to OpenVZ but we had an IP bound to eth0:0 and after I removed it the traffic jumped to 140Mbps. I tested it on a KVM VPS (installed OpenVZ and set it up just like our other OpenVZ nodes) with no luck either. We used to think the problem was ARP related but this is not the case since flushing the ARP tables on the server or network devices have no effect.

I am willing to pay $200 via Paypal if somebody can explain it and provide a solution that does not involve purchasing additional equipment.
Click to expand...
A quote from the OP with the information pertinent in BOLD.

Veddy
 
R

rds100

New Member
Verified Provider
@vedalken254 there is no such thing as virtual MAC or separate MAC for an ip alias. It's the same MAC address as eth0, just an additional IP added.
 
V

vedalken254

New Member
rds100 said:
@vedalken254 there is no such thing as virtual MAC or separate MAC for an ip alias. It's the same MAC address as eth0, just an additional IP added.
Click to expand...
I'm not going to continue to argue with you as you keep failing to read critical portions of what my posts say (INCLUDING THE EDITS!) and thus, it would fall on deaf ears.
 
R

rds100

New Member
Verified Provider
Yes, i saw your bold text. However KuJoe also said he doesn't own the switches / routers and doesn't have access to them.

So someone told him it's not in a table on the router? Which table did that someone mean, the routing table? Did that someone ckeck the ARP table? Who knows.
 
V

vedalken254

New Member
rds100 said:
Yes, i saw your bold text. However KuJoe also said he doesn't own the switches / routers and doesn't have access to them.

So someone told him it's not in a table on the router? Which table did that someone mean, the routing table? Did that someone ckeck the ARP table? Who knows.
Click to expand...
Considering the fact that they said it wasn't present in ANY tables, wouldn't that suffice to mean they checked them all?
 
R

rds100

New Member
Verified Provider
Yes, it would also make sense that the DC should be able to figure out what is going on and advise the customer how to fix it. Obviously it didn't happen. Who knows what low pay employee is answering the tickets and if he knows where to look?
 
F

Flapadar

Member
Verified Provider
Here's a hacky (possible) fix

iptables -t mangle -I PREROUTING -i ethx  -j TTL --ttl-set 1

Not a great idea to do for all traffic so you may want to specificly match the problematic IP. Additionally you want to only choose inbound traffic (outbound traffic will still need a high TTL)
 
Last edited by a moderator:
KuJoe

KuJoe

Well-Known Member
Verified Provider
rds100 said:
This seems like the same packet cycling until the TTL runs out.

Try tcpdump -v -v -v to see if these packets differ by the ttl (decreasing with each next).

What happens according to me:

- Rogue random packet comes from the internet (port scans, whatever)

- Your router, L3 switch or whatever is the gateway device to your network has a cached ARP packet for this IP and is sending the packet to the MAC address of your server.

- Your server is receiving the packet (because it's destined to it's MAC address), but doesn't recognize the destination IP as it's own so has to forward the packet.

- The packet is forwarded according to the routing table (i.e. to the default gateway), and the TTL is decreased by 1

- The default gateway (your router / L3 switch / whatever) receives the packet and routes it back to your server (again decreasing the TTL by 1)

- repeat until the TTL expires (reaches zero).

Do you have control over the router (or whatever the gateway to your network is)? Can you check or clear the ARP cache there?
Click to expand...
We have had the data center clear the ARP cache on their devices and it does not stop the traffic. We also have each network set the ARP timeout to 5 minutes and this traffic will go on indefinitely until the external source IP stops.

I'll try that tcpdump command next time this happens.
 
KuJoe

KuJoe

Well-Known Member
Verified Provider
OMG, you guys are amazing! Thanks for @rds100 for putting me on the right track and thanks for @Flapadar for getting me even closer I have found the problem and have found 2 solutions for it (both suggested by @rds100 and @Flapadar). The cause of the problem is due to my own lack of iptables knowledge and me blindly running a command I wasn't familiar with over a year ago. The problem was I ran an iptables command to remove the host node's IP from traceroutes by increasing the TTL, unfortunately this command was in the /etc/rc.local file so every reboot it would add the same iptables rule so for this example, the node would forever increase the TTL so I could turn a single ICMP packet into a 100Mbps flood (I tested this tonight and it went from a few Bps to 110Mbps with a single ICMP if I removed the IP while the continuous ping was going).

The iptables command @rds100 provided did work and was a valid solution but a simple iptables -t mangle -F fixed the problem immediately also. The TTL conversation put me on the right path, I ran the tcpdump -v -v -v and saw the TTL alternating between 253 and 254. Then I saw @Flapadar's post and noticed that the iptables command he posted was almost identical to the command I ran over a year ago. After inspecting the command I noticed that it adjusted the TTL and then after viewing the mangle table I saw where the problem was.

Also a big thank you to @vedalken254 for continuing the discussion in this thread while I was sleeping and keeping it visible. Your posts mirrored the exact thoughts my partner and I had but unfortunately it wasn't MAC related. :)

@Flapadar@rds100 PM me your Paypal accounts and I'll send you each $100 between you since I wouldn't have come up with the answer without both of your posts.
 
Hxxx

Hxxx

Active Member
rds100 said:
@KuJoe if i take your money i'd have to do a lot of paperwork for taxes, etc... not worth the hassle for me :)

I didn't participate here for the money anyway. Maybe some day you can help me on some different matter where i need advise, who knows.
Click to expand...
I don't see how 100 dollars will create a hassle for you in taxes. But technically he is not paying you for work (yeah figure). Basically he is being grateful.

Is just 100 dollars, not 100,000 or 1,000.
 
R

rds100

New Member
Verified Provider
@ no matter if it's $1, $100 or $10000 it is income and i would have to fill tax declaration and the hassle and paperwork is the same, regardless of the amount. Right now i don't have to fill and submit yearly tax declarations, because my only income is from salary from my employer and in this case submitting tax declaration is not necessary.
 
Last edited by a moderator:
Hxxx

Hxxx

Active Member
rds100 said:
@ no matter if it's $1, $100 or $10000 it is income and i would have to fill tax declaration and the hassle and paperwork is the same, regardless of the amount. Right now i don't have to fill and submit yearly tax declarations, because my only income is from salary from my employer and in this case submitting tax declaration is not necessary.
Click to expand...
ok as you wish. As far as I know, it doesnt matter if you work for a company or for yourself, you always have to fill your income tax, unless the amount is so small...
 
R

rds100

New Member
Verified Provider
@ not here. When you work for a company (and no other incomes) you don't need to fill tax declarations, the employer automatically deducts the tax from your salary and pays it every month.

Besides i don't need the money. I still make more than my wife spends ;-)

And i haven't worked hard enough to earn it :) I am just helping a fellow with some brainstorming.
 
You must log in or register to reply here.
Top
amuck-landowner