# HOWTO: Stop NTP amplification attacks from reaching your nodes!



## Francisco (Feb 18, 2014)

Hello everyone,

***If you share this information on your blog, forum, etc, be kind and please link back to this topic!***

Normally I'm not one to share stallion code, but after a discussion with a couple staffers we came to the conclusion that the following work must be made public for the 'greater good' and all that righteous crap.

*What this does*

The following blocks NTP *monlist* packets at the *node* (or router level if you're using a linux based setup), before they ever get to your customers. This means that it provides preemptive filtering, instead of after-the-fact-oh-god-my-bandwidth-bills. Stopping NTP amplification floods before the user gets them was the only way for us morally address users from being used in NTP floods be it now or later on.

*What this doesn't do*

This does *not* patch the users configuration files by any means. This is entirely node side done with iptables. You should still make it an effort to inform your customers about the dangers of using a bad version of NTP.

*Lets get started!*

First, you must add the following entry to your */etc/sysctl.conf*. This makes it so all packets sent over a bridge (for XEN & KVM based VM's) are also filtered.
 


net.bridge.bridge-nf-call-iptables = 1

Once this is done, apply the changes
 


sysctl -p

The following rule is what does all the magic. You'll want to put this in /etc/rc.local above the *exit 0* so it gets applied on reboot. You should also look at using *iptables-save* as well.
 


iptables -I FORWARD -p udp --dport 123 -m u32 --u32 "0x1C=0x1700032a && 0x20=0x00000000" -m comment --comment "NTP amplification packets" -j DROP

You can change the chain from *FORWARD* to *INPUT* in the off chance that you want to use this inside a VPS or something like that. It'd be smarter to simply ACL monlist or upgrade your version, but to each their own.

You should feel no performance impact from this rule being in place. Your node will still be smacked with the packets, but nothing will be sent out.

For obvious reasons, I won't be talking to Phill about including this in his node side SolusVM code, but if someone wishes to point him this way, they have my permission to include this.

For the greater good,

Francisco
Your friendly neighborhood hairyman
 
 
_EDIT: Added the below quoted response upon request of OP. -MD_
 



Magiobiwan said:


> I modified Mun's script slightly to make it also edit /etc/rc.local. The new script version can be found at...
> 
> 
> ```
> ...


----------



## bzImage (Feb 18, 2014)




----------



## Kakashi (Feb 18, 2014)

Very nice, appreciate the help.


----------



## splitice (Feb 18, 2014)

Good work Fran / Delta,

When doing this for the sake of your customers flexibility (and sanity when debugging) either ensure your conntrack size is suitably large or disable connection tracking for these packets when enabling iptables for a KVM emulated network interface. Most people assume that a KVM virtual nic is the same as a bare metal one and not limited in terms of connections.

Disabling conntrack can be done with a rule in the PREROUTING chain -j NOTRACK or by not having the module loaded / compiled in.


----------



## Mun (Feb 18, 2014)

I think I wrote up a bash script that should automate the process. Here is the source!

http://cdn.content-network.net/Mun/apps/frantp/0.1/source.txt

You should be able to run it via:


wget http://www.cdn.content-network.net/Mun/apps/frantp/0.1/script.sh -O - | bash

or for https://

wget https://www.cdn.content-network.net/Mun/apps/frantp/0.1/script.sh -O - | bash



I am also looking for suggestions on how to improve the code.

Please note, that unless you run this as a rc.local task the IPtables will be flushed on reboot! I suggest you save the file to a local directory, chmod +x it, and then make a rc.local task.

Please note I have tested it on my dedicated server, but not on a VPS node, can someone please confirm it works?

Mun


----------



## HostUS-Alexander (Feb 18, 2014)

<3


----------



## kaniini (Feb 18, 2014)

If you want the inverse of this (to filter the crap out that you're receiving), simply block UDP packets to port 123 which are 468 octets long. You can do this with a route-map on most routers or a policer.


----------



## Mun (Feb 18, 2014)

kaniini said:


> If you want the inverse of this (to filter the crap out that you're receiving), simply block UDP packets to port 123 which are 468 octets long. You can do this with a route-map on most routers or a policer.


Should I add this to the script, or it that bad? If so, then what would the iptables command look like?

Mun


----------



## mcmyhost (Feb 18, 2014)

What about IPv6?


----------



## Mun (Feb 18, 2014)

Snipped just so now one does anything stupid.


----------



## Francisco (Feb 18, 2014)

Mun said:


> ip6tables -I FORWARD -p udp --dport 123 -m u32 --u32 "0x1C=0x1700032a && 0x20=0x00000000" -m comment --comment "NTP amplification packets" -j DROP
> I assume.. don't quote me on it...


This is most likely incorrect.

I'll poke around with tcpdump later and see what it actually is.

Francisco


----------



## Francisco (Feb 18, 2014)

kaniini said:


> If you want the inverse of this (to filter the crap out that you're receiving), simply block UDP packets to port 123 which are 468 octets long. You can do this with a route-map on most routers or a policer.


For sure!

With this iptables rule, though, it allows users that are renting all their gear and don't have switch ACL access to defend from it as well.

Francisco


----------



## Francisco (Feb 18, 2014)

splitice said:


> Good work Fran / Delta,
> 
> When doing this for the sake of your customers flexibility (and sanity when debugging) either ensure your conntrack size is suitably large or disable connection tracking for these packets when enabling iptables for a KVM emulated network interface. Most people assume that a KVM virtual nic is the same as a bare metal one and not limited in terms of connections.
> 
> Disabling conntrack can be done with a rule in the PREROUTING chain -j NOTRACK or by not having the module loaded / compiled in.


We've had no performance issues to date with it. We actually had a few users get banned by autonull because people were sending so many packets at them to reflect.

Francisco


----------



## Mun (Feb 18, 2014)

kaniini said:


> If you want the inverse of this (to filter the crap out that you're receiving), simply block UDP packets to port 123 which are 468 octets long. You can do this with a route-map on most routers or a policer.


When I was reading this, I was thinking about it in a whole different way.... oppps.


----------



## FLDataTeK (Feb 18, 2014)

Thank you for figuring out how to block it at the node level I am sure it will help everyone rest better at night knowing that their nodes won't be assisting in the DDOS...


----------



## eva2000 (Feb 18, 2014)

interesting info posted at http://www.micron21.com/ddos-ntp.php


----------



## Francisco (Feb 18, 2014)

Same concept just I hope they aren't doing literal string matches 

Francisco


----------



## Mun (Feb 18, 2014)

@Francisco, do you see anything I should add to my script?


----------



## Francisco (Feb 18, 2014)

Mun said:


> @Francisco, do you see anything I should add to my script?


Looks good to me.

Francisco


----------



## Mun (Feb 18, 2014)

Francisco said:


> Looks good to me.
> 
> 
> Francisco



KK, think I will make another one without the bridge adapter for people whom wont be using it on a VPS setup.


----------



## Francisco (Feb 18, 2014)

Mun said:


> KK, think I will make another one without the bridge adapter for people whom wont be using it on a VPS setup.


you could just check the output of brctl list and see if they have a bridge or not?

If they don't have a bridge (openvz) then the entry won't 'do' anything wrong, it won't even load since the sysctl params won't exist.

Francisco


----------



## Magiobiwan (Feb 18, 2014)

Fran, you are awesome. As it happens your posting this coincided with several of our nodes being hammered from outbound attacks thanks to misconfigured NTP servers. I feel obliged to purchase a VPS from you now.


----------



## Magiobiwan (Feb 18, 2014)

I modified Mun's script slightly to make it also edit /etc/rc.local. The new script version can be found at...


wget http://darkrai.unovarpgnet.net/antintp.sh -O - | bash

No HTTPS for it; my server in SEA doesn't have a cert. You could download over HTTPS I suppose, it'd just complain it's invalid.


----------



## TruvisT (Feb 19, 2014)

Good topic. Referenced


----------



## Francisco (Feb 19, 2014)

TruvisT said:


> Good topic. Referenced


Thanks!

Anyone had contact with solusvm or is that a lost cause?

Francisco


----------



## Virtovo (Feb 19, 2014)

Francisco said:


> Thanks!
> 
> 
> Anyone had contact with solusvm or is that a lost cause?
> ...


I've dropped them a ticket as you've specifically referenced it.  I have no prior experience with SolusVM; however they are quite quickly reversing bad things I have read about them.  This week they have offered swift resolution to two issues I had with one of them being a feature added that is in the latest beta.  Still not going to let IPv6 handling go.  Wish there was a panel that worked and assigned /64s!


----------



## Francisco (Feb 19, 2014)

Virtovo said:


> I've dropped them a ticket as you've specifically referenced it.  I have no prior experience with SolusVM; however they are quite quickly reversing bad things I have read about them.  This week they have offered swift resolution to two issues I had with one of them being a feature added that is in the latest beta.  Still not going to let IPv6 handling go.  Wish there was a panel that worked and assigned /64s!


K, cool. Sorry for coming off as nagging, but it'd likely chop down the amount of bandwidth involved in floods if they merged it.

Francisco


----------



## Virtovo (Feb 19, 2014)

Francisco said:


> K, cool. Sorry for coming off as nagging, but it'd likely chop down the amount of bandwidth involved in floods if they merged it.
> 
> 
> Francisco


No of course.  I didn't realise the the history you had with SolusVM.  I have since read some threads about it.


----------



## Francisco (Feb 19, 2014)

Virtovo said:


> No of course.  I didn't realise the the history you had with SolusVM.  I have since read some threads about it.


There's more that isn't public ;p

With that being said, this thread is about something much bigger.

Francisco


----------



## Virtovo (Feb 19, 2014)

Francisco said:


> There's more that isn't public ;p
> 
> 
> With that being said, this thread is about something much bigger.
> ...


Agreed.  Are you spreading the word further afield?


----------



## Francisco (Feb 19, 2014)

Virtovo said:


> Agreed.  Are you spreading the word further afield?


As I can but WHT doesn't seem to give much of a crap about it.

I have a lot of other projects going on right now so I can't really pump tons

of time into awareness on this.

Maybe I should try to get the cloudflare guy to retweet this link and or write

a small blurb about it? I'll tweet him and see what happens.

Francisco


----------



## Virtovo (Feb 19, 2014)

Francisco said:


> As I can but WHT doesn't seem to give much of a crap about it.
> 
> 
> I have a lot of other projects going on right now so I can't really pump tons
> ...


Maybe drop a message to the NTF: http://nwtime.org/ they operate http://openntpproject.org/


----------



## Francisco (Feb 19, 2014)

Virtovo said:


> Maybe drop a message to the NTF: http://nwtime.org/ they operate http://openntpproject.org/


Done as well 

Francisco


----------



## mcmyhost (Feb 19, 2014)

I've been looking around tcpdump and cannot find the correct way to filter IPv6.

Perhaps you've had better luck?


----------



## Francisco (Feb 19, 2014)

mcmyhost said:


> I've been looking around tcpdump and cannot find the correct way to filter IPv6.
> 
> Perhaps you've had better luck?


I've not had time to look into the v6 packets for it.

I've been caught up with backups & working on an autonull for fiberhub.

Francisco


----------



## Mun (Feb 20, 2014)

Does all OS have brctl as a command for brige-utils? I am trying to test if the bridge exists via the "addbr" command.

Mun

Update:

https://cdn.content-network.net/Mun/apps/frantp/0.2/source.txt <-- here is the new source. I added the below code to check to see if brctl is installed, and if so then run the


"net.bridge.bridge-nf-call-iptables = 1" echo into sysctl.conf

```
if brctl --help | grep -q "addbr"; then
    if grep -q "net.bridge.bridge-nf-call-iptables = 1" "/etc/sysctl.conf"; then
    echo "Sysctl already done!"
    else
    echo "net.bridge.bridge-nf-call-iptables = 1" >> /etc/sysctl.conf
    sysctl -p
    fi
else
    echo "No bridge installed..."
fi
```
I also added magiobiwans rc.local code as well as adding his script to my server, so that it could be accessed over https://.


```
For https://
wget https://cdn.content-network.net/Mun/apps/frantp/0.2/script.sh -O - | bash -

for http://
wget http://cdn.content-network.net/Mun/apps/frantp/0.2/script.sh -O - | bash -

for Magiobiwan's script over https://
wget https://cdn.content-network.net/Mun/apps/frantp/magiobiwan/1.2/script.sh -O - | bash -
```


----------



## Mun (Feb 20, 2014)

New Version 0.5! Checks to see if Magiobiwan's rc.local code chunk is already installed, and if so skips. This prevents duplicates in rc.local!

Source: https://cdn.content-network.net/Mun/apps/frantp/0.5/source.txt


For https://
wget https://cdn.content-network.net/Mun/apps/frantp/0.5/script.sh -O - | bash -

for http://
wget http://cdn.content-network.net/Mun/apps/frantp/0.5/script.sh -O - | bash -

Mun


----------



## Magiobiwan (Feb 20, 2014)

Now all we need to make it do is solve world hunger and create world peace!


----------



## mcmyhost (Feb 20, 2014)

Magiobiwan said:


> Now all we need to make it do is solve world hunger and create world peace!


$worldpeace = false;
$worldhunger = false;

if $worldpeace = false {
$worldpeace = true;
}

if $worldhunger = false {
$worldhunger = true;
}

Simple!


----------



## Francisco (Feb 20, 2014)

if brctl --help | grep -q "addbr"; thenbrctl can be installed and will answer even if no bridge exists.
You'd be better off checking if /proc/sys/net/bridge-nf-call-iptables exists or not.

If it does, a bridge is active.

Francisco


----------



## Mun (Feb 20, 2014)

Francisco said:


> if brctl --help | grep -q "addbr"; then
> brctl can be installed and will answer even if no bridge exists.
> 
> You'd be better off checking if /proc/sys/net/bridge-nf-call-iptables exists or not.
> ...



Alright, ill start working on that now 

Thanks


----------



## Mun (Feb 20, 2014)

As per @Francisco's suggestion I have changed how I lookup a bridge.

Source: http://cdn.content-network.net/Mun/apps/frantp/0.6a/source.txt


Http://

wget http://cdn.content-network.net/Mun/apps/frantp/0.6a/script.sh -O - | bash -


https://

wget https://cdn.content-network.net/Mun/apps/frantp/0.6a/script.sh -O - | bash -

Code change:


if [ -a "/proc/sys/net/bridge/bridge-nf-call-iptables" ]; then

# 
# This allows us to check if the bridge exists in a more appropriate way.
#

Anymore suggestions?

Mun


----------



## bzImage (Feb 21, 2014)

First they came for the Socialists, and I did not speak out-- Because I was not a Socialist.
Then they came for the Trade Unionists, and I did not speak out-- Because I was not a Trade Unionist.
Then they came for the Jews, and I did not speak out-- Because I was not a Jew.
Then they came for me--and there was no one left to speak for me.
 

Instead of looking for the next get rich quick scheme remember when a good portion of the net went dark because this exploit was used to take level3 offline. Food for thought.


----------



## Virtovo (Feb 22, 2014)

Has anyone seen any issues with conntrack as a result of this fix?


----------



## Francisco (Feb 22, 2014)

Virtovo said:


> Has anyone seen any issues with conntrack as a result of this fix?


Not on our nodes, no.

Francisco


----------



## TrentaHost (Feb 26, 2014)

Thank You for sharing this Francisco


----------



## drmike (Feb 26, 2014)

So help a sad person out...

CentOS 5...


sysctl -p
net.ipv4.ip_forward = 1
net.ipv4.ip_forward = 1
net.ipv6.conf.default.forwarding = 1
net.ipv6.conf.all.forwarding = 1
net.ipv4.conf.default.proxy_arp = 0
net.ipv4.conf.all.rp_filter = 1
kernel.sysrq = 1
net.ipv4.conf.default.send_redirects = 1
net.ipv4.conf.all.send_redirects = 0
kernel.panic = 10
net.bridge.bridge-nf-call-iptables = 1



*error: "net.bridge.bridge-nf-call-iptables" is an unknown key*

So is the issue the CentOS version?


----------



## Francisco (Feb 26, 2014)

drmike said:


> So help a sad person out...
> 
> CentOS 5...
> 
> ...


Is it an OpenVZ node or a KVM?

sysctl -a | grep bridge

Francisco


----------



## Magiobiwan (Feb 26, 2014)

If there's not a bridge set up, then that key won't exist. On OpenVZ nodes, there shouldn't be a bridge.


----------



## rapidnode (Feb 27, 2014)

Thanks for sharing this! Also, make sure your conntrack tables don't fill up =)


----------



## Mun (Feb 27, 2014)

drmike said:


> So help a sad person out...
> 
> CentOS 5...
> 
> ...



What version are you using to run this, I assume you are using the script. As of current Magiobiwans script simply throws it in place, as did my initial versions. If you use 0.6a You shouldn't have that issue. I also suggest you remove the 'net.bridge.bridge-nf-call-iptables = 1' from your /etc/systl.conf if you aren't using openVZ as you don't need it.

Mun


----------



## splitice (Mar 27, 2014)

A late addition to the thread:


iptables -A FORWARD ! -f -p udp -m multiport --ports 123 -m u32 --u32 "0>>22&[email protected]&0xFF=42" -j DROP
Will block all UDP monlist packet IN or OUT allowing you to also stop any MONLIST packets that may leak through hardware protection (of course this is not a substitute for good protection).

A slightly altered variant of this rule has been tested with a stresser


----------



## key900 (Mar 13, 2015)

Great work seems good way.


----------



## X3host (May 18, 2015)

Is this work with openvz OS ??


----------



## Francisco (May 19, 2015)

CoMBoZo said:


> Is this work with openvz OS ??


This just stops your VM's from being used to DDOS other people, it won't stop NTP floods from hitting you straight in the face.

Francisco


----------



## VPSclub (Feb 13, 2016)

Helpful tutorial, thanks.


Is there any way to stop DNS amplification attack? Any tutorial on this, would be highly appreciated.


----------

