Hardening my server / Reality check

[stim]

Hi,

I just got a new server from a reputable provider I've used for some time . All is well, and I'm set-up with Ubuntu14.04.

My first steps when getting a new server, after updating it, is to implement some basic security:

1. Disable root.

2. Enable only one user, added to sudoers (I am the only user)

3. Change ssh port.

4. Install UFW and open only the ssh port, 80, 443.

5  Install and configure Fail2Ban (I always used Denhyhosts, but it's not supported in Ubuntu14.04) 

6 Turned off all unnecessary services listed by :


sudo sysv-rc-conf --list | grep '3n'
I also have PubKey authentication, however, it is unfortunately necessary to allow password login for my one account. To my less-than-wizened eye this would seem to be the weak point of my setup (even if the PW is a long one), nevertheless I think that, overall, I'm secure enough to easily thwart all but the most determined attacker.

Checking the syslog confirms this. I have logging set to low, nevertheless I am seeing a lot of activity - more so than other servers I have used.

Using this one-liner to display a numbered list of unique IP numbers blocked by UFW...


sudo grep -o 'SRC=[0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}' /var/log/syslog |sort | uniq | sed -e 's/SRC=//g' | cat -n 
... I am getting approx 1000 unique IP blocks per day, with many repeated attempts.  Since >90% are Chinese, trying to connect on 22 or some random port, I've looked into blocking by country, but this seems like more work than needed. If my firewall is working OK I should have nothing to worry about, right?

My question is: 

Is there a level of UFW blocks per minute at which I should become concerned and, most importantly does this have an adverse effect on the performance of my server?

Are there other measures I should consider?

This is not a critical server and I'm not super-paranoid, more curious about what is perceived as normal these days.

Thanks in advance for your sage advice.

 

[fixidixi]

you could ask your provider to block those CN ips if they provide such option. of course it affects your performance as your firewall has to drop connections..

 

[Steven F]

@MartinD, I seem to remember you having similar problems from CN IPs, maybe you can comment on how you dealt with it?

 

[MartinD]

Just custom scripts.. and when XX amount of a certain range were identified the whole subnet gets blocked

 

[stim]

@fixidixi - Thanks. I think this is the easiest solution, and I'm sure they will oblige.

However I'm going to hold-off for now and use it as a learning opportunity. I've already started scripting.

@MartinD - I don't even know where to begin with subnet blocking, so I'll be on Google for the evening it seems

So far I would have thought that such a thing was easy. The suggestions I have seen do far involve either loading a huge list of UFW rules (which seems pointless if UFW is already blocking), or the use of geoip databases.

I found a nice Ruby gem that uses a geoip db to expand the details from each ip. I'm going to make a webpage with some live statistics and just observe the phenomenon for a while. I think the current levels are more than manageable. When my web-app is fully up-and running I will finally restrict to only EU and US IPs.

I feel some degree of comfort that nobody castigated me for my security setup. 

Cheers

 

[Abdussamad]

You shouldn't be blocking at all. Since you have a strong SSH password they are basically hitting a brick wall. Let them bang their heads against it as much as they like. It doesn't hurt you. What does hurt you is blocking 1000s of IPs like this.

If you are really worried about this you can restrict ssh access to your IP address only. This is a whitelist instead of a blacklist and it won't hurt server performance at all.

Finally if you absolutely must blacklist then use ipset. That is the most efficient way of blocking entire countries.

 

[Munzy]

you can use enjen.net/asn-blocklist to create block lists to block everything that is coming from those unreputable ASNs.

 

[TurnkeyInternet]

It's brutal the volume of brute force ips you will find on a given moment originating from non-US (CN is the #1 i've noticed too).  SSH isn't the only spot they are brute forcing, they are checking for every finger print / entry point (MySQL, and of course probing web etc).

Honestly, since its so distributed its not as simple as block all traffic from CN based on IP's (you can use BGP annoucements to find basically the troublesome originators and what ip blocks are represented in the CN space you have issues with pretty easily, but its a very lengthy amount of IP ranges you have to ban).  You are doing the right stuff already with best security practices - my personal favorite is CSF/LFD as it catches all the finger-print scanner bots after a few connetions to lock them down, and clears the ip blacklist over time.

 

[drmike]

You shouldn't be blocking at all. Since you have a strong SSH password they are basically hitting a brick wall. Let them bang their heads against it as much as they like. It doesn't hurt you. What does hurt you is blocking 1000s of IPs like this.

If you are really worried about this you can restrict ssh access to your IP address only. This is a whitelist instead of a blacklist and it won't hurt server performance at all.

Finally if you absolutely must blacklist then use ipset. That is the most efficient way of blocking entire countries.

Tell a provider with a busy VPS server node this.  The part about SSH attempts not hurting you.

More IPs routed to a box = multiplied effect.

It really can and does create a traffic jam, deplete random pool and creates artificial slagging on the server.

Dealing with these IPs and attempts should be something any decent sized provider already does by finding such requests where they shouldn't be (i.e. honeypot).  Then applying a company wide block with some form of history and escalation.

For an end customer though, seems like a common approach to hardening things.  Surely more advanced things to add on.  Hopefully some of the folks in the trenches are willing to share.

 

[stim]

Thanks again for the advice above. 

As far as I can gather, I'm seeing 'normal', and current volume is perfectly manageable (it even seems to have dropped-off).

For fun, I knocked-up a little script to scan for UFW blocks and pinpoint the source ip on Google Maps.

Not accurate in all cases of course - but there are some hilarious results.

If nothing else some free data to play with.

 

[Aldryic C'boas]

Out of curiosity.. what's the one service preventing you from locking SSH down to key access only?  If it's something like FTP you could always go the SCP/SFTP route.

 

[stim]

Out of curiosity.. what's the one service preventing you from locking SSH down to key access only?  If it's something like FTP you could always go the SCP/SFTP route.

It's a proprietary Windows client. I've tried using Paegent to push the keys to it but as yet no success.

Indeed if there is another way to get windows applications to pull the keys I'd love to know... 

 

[fixidixi]

@stim:

I dont exactly get it why would you be locked down to an ssh client.

there are a whole bunch of putty forks out there with tabbed option or my personal favourite: puttytray. well i have only 20 connections or so its all well under control i guess it would be too much at a certain point. but with "store in file" option puttytray is pretty portable and thats a huge advantage for me

 

[Wild1145]

Wildcard banning IP's is not often an amazing idea... Often attacks will come from multiple ranges where only maybe 10 IP's are actually doing anything to harm your server... You could change the SSH port to something stupid that people wont guess, but from my understanding there is some software that does require it to be on port 22. Plus you can run into firewall issues on networks if you are not connecting from home (Eg Work)

 

[stim]

@fixidixi

No, it's a little more complex than that. In the end I have come up with a simple stopgap solution where I can temporarily enable passwords, then re-disable them after the client connects (it's not a dedicated ssh client, but works over sftp).

Also, I have a very nice and portable set-up already using MultiTabbed Putty and Kitty

@Wild1145

SSH port is already changed. Nobody is getting through the firewall. If they do find my ssh port they are welcome to try.

From the statistics I've gathered so, I am clearly inclined to block all Chinese IP addresses in future.

I've also fired-off a few angry e-mails to hosting providers facilitating the most persistent IPs. Surely they must notice the drain.

The most popular port so far is 9064/TCP by a mile, but mostly via only 4 Chinese ip addresses.

After 2 days, the scores are:

9064/TCP 653 blocks
8118/TCP 288 blocks
23/TCP 233 blocks
43767/UDP 181 blocks
22/TCP 80 blocks
5060/UDP 68 blocks
1433/TCP 54 blocks
443/TCP 54 blocks
8080/TCP 52 blocks
3389/TCP 42 blocks
13814/UDP 40 blocks
3128/TCP 37 blocks
1900/UDP 36 blocks
8088/TCP 26 blocks
3306/TCP 25 blocks
8123/TCP 25 blocks
25/TCP 23 blocks
110/TCP 19 blocks
53/UDP 19 blocks
5900/TCP 17 blocks
8081/TCP 15 blocks
123/UDP 13 blocks
4899/TCP 13 blocks
81/TCP 13 blocks
19/UDP 11 blocks
9000/TCP 11 blocks
135/TCP 10 blocks
21320/TCP 10 blocks
8000/TCP 10 blocks
1080/TCP 9 blocks
21/TCP 9 blocks
83/TCP 8 blocks
8090/TCP 7 blocks
84/TCP 7 blocks
8585/TCP 7 blocks
8888/TCP 7 blocks
9200/TCP 7 blocks
0/TCP 6 blocks

 

[MartinD]

That looks about right.

Have you looked at using IPSet for blocks across all your machines?

 

[stim]

That looks about right.

Have you looked at using IPSet for blocks across all your machines?

Yeah, I have. And my provider will do it for me if I need. I'm just going to leave to for a while, observe and worry about it when I'm production-ready. At that stage I will probably do a fresh install.

I'll develop my script some more and publish it. It's a nice little eye-opener.

Cheers for all the help and reassurance  

 

[Abdussamad]

What about whitelisting your IPs? Or using a jump host and whitelisting that alone.

OpenSSH/Cookbook/Proxies and Jump Hosts - Wikibooks, open books for an open world

 

[MartinD]

Yeah, I have. And my provider will do it for me if I need. I'm just going to leave to for a while, observe and worry about it when I'm production-ready. At that stage I will probably do a fresh install.

I'll develop my script some more and publish it. It's a nice little eye-opener.

Cheers for all the help and reassurance

I've just had a customer (literally in the past 10 minutes) asking me to help with blocking China from their VM for this very reason! You're certainly not alone with the problem!

 

[stim]

Hi,

Back again with a related question:

Two of the most scanned ports on my server are:

9064/TCP 3776 blocks
43767/UDP 805 blocks

9064 is constantly being scanned by the same Chinese subnet, accounting for over one third of all connection attempts.

Similarly, I've noticed 43767 getting absolutely hammered since yesterday, but this time from multiple sources.

Both ports are closed, and a Google search suggests that they are generally unassigned.

So why should these ports attract such attention? 

Any thoughts//experiences?

Cheers