# Hardening my server / Reality check



## stim (Nov 17, 2014)

Hi,

I just got a new server from a reputable provider I've used for some time . All is well, and I'm set-up with Ubuntu14.04.

My first steps when getting a new server, after updating it, is to implement some basic security:

1. Disable root.

2. Enable only one user, added to sudoers (I am the only user)

3. Change ssh port.

4. Install UFW and open only the ssh port, 80, 443.

5  Install and configure Fail2Ban (I always used Denhyhosts, but it's not supported in Ubuntu14.04) 

6 Turned off all unnecessary services listed by :


sudo sysv-rc-conf --list | grep '3n'
I also have PubKey authentication, however, it is unfortunately necessary to allow password login for my one account. To my less-than-wizened eye this would seem to be the weak point of my setup (even if the PW is a long one), nevertheless I think that, overall, I'm secure enough to easily thwart all but the most determined attacker.

Checking the syslog confirms this. I have logging set to low, nevertheless I am seeing a lot of activity - more so than other servers I have used.

Using this one-liner to display a numbered list of unique IP numbers blocked by UFW...


sudo grep -o 'SRC=[0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}' /var/log/syslog |sort | uniq | sed -e 's/SRC=//g' | cat -n 
... I am getting approx 1000 unique IP blocks per day, with many repeated attempts.  Since >90% are Chinese, trying to connect on 22 or some random port, I've looked into blocking by country, but this seems like more work than needed. If my firewall is working OK I should have nothing to worry about, right?

My question is: 

Is there a level of UFW blocks per minute at which I should become concerned and, most importantly does this have an adverse effect on the performance of my server?

Are there other measures I should consider?

This is not a critical server and I'm not super-paranoid, more curious about what is perceived as normal these days.

Thanks in advance for your sage advice.


----------



## fixidixi (Nov 17, 2014)

you could ask your provider to block those CN ips if they provide such option. of course it affects your performance as your firewall has to drop connections..


----------



## Steven F (Nov 17, 2014)

@MartinD, I seem to remember you having similar problems from CN IPs, maybe you can comment on how you dealt with it?


----------



## MartinD (Nov 17, 2014)

Just custom scripts.. and when XX amount of a certain range were identified the whole subnet gets blocked


----------



## stim (Nov 17, 2014)

@fixidixi - Thanks. I think this is the easiest solution, and I'm sure they will oblige.

However I'm going to hold-off for now and use it as a learning opportunity. I've already started scripting.

@MartinD - I don't even know where to begin with subnet blocking, so I'll be on Google for the evening it seems 

So far I would have thought that such a thing was easy. The suggestions I have seen do far involve either loading a huge list of UFW rules (which seems pointless if UFW is already blocking), or the use of geoip databases.

I found a nice Ruby gem that uses a geoip db to expand the details from each ip. I'm going to make a webpage with some live statistics and just observe the phenomenon for a while. I think the current levels are more than manageable. When my web-app is fully up-and running I will finally restrict to only EU and US IPs.

I feel some degree of comfort that nobody castigated me for my security setup. 

Cheers


----------



## Abdussamad (Nov 17, 2014)

You shouldn't be blocking at all. Since you have a strong SSH password they are basically hitting a brick wall. Let them bang their heads against it as much as they like. It doesn't hurt you. What does hurt you is blocking 1000s of IPs like this.

If you are really worried about this you can restrict ssh access to your IP address only. This is a whitelist instead of a blacklist and it won't hurt server performance at all.

Finally if you absolutely must blacklist then use ipset. That is the most efficient way of blocking entire countries.


----------



## Munzy (Nov 17, 2014)

you can use enjen.net/asn-blocklist to create block lists to block everything that is coming from those unreputable ASNs.


----------



## TurnkeyInternet (Nov 17, 2014)

It's brutal the volume of brute force ips you will find on a given moment originating from non-US (CN is the #1 i've noticed too).  SSH isn't the only spot they are brute forcing, they are checking for every finger print / entry point (MySQL, and of course probing web etc).

Honestly, since its so distributed its not as simple as block all traffic from CN based on IP's (you can use BGP annoucements to find basically the troublesome originators and what ip blocks are represented in the CN space you have issues with pretty easily, but its a very lengthy amount of IP ranges you have to ban).  You are doing the right stuff already with best security practices - my personal favorite is CSF/LFD as it catches all the finger-print scanner bots after a few connetions to lock them down, and clears the ip blacklist over time.


----------



## drmike (Nov 17, 2014)

Abdussamad said:


> You shouldn't be blocking at all. Since you have a strong SSH password they are basically hitting a brick wall. Let them bang their heads against it as much as they like. It doesn't hurt you. What does hurt you is blocking 1000s of IPs like this.
> 
> If you are really worried about this you can restrict ssh access to your IP address only. This is a whitelist instead of a blacklist and it won't hurt server performance at all.
> 
> Finally if you absolutely must blacklist then use ipset. That is the most efficient way of blocking entire countries.


Tell a provider with a busy VPS server node this.  The part about SSH attempts not hurting you.

More IPs routed to a box = multiplied effect.

It really can and does create a traffic jam, deplete random pool and creates artificial slagging on the server.

Dealing with these IPs and attempts should be something any decent sized provider already does by finding such requests where they shouldn't be (i.e. honeypot).  Then applying a company wide block with some form of history and escalation.

For an end customer though, seems like a common approach to hardening things.  Surely more advanced things to add on.  Hopefully some of the folks in the trenches are willing to share.


----------



## stim (Nov 18, 2014)

Thanks again for the advice above. 

As far as I can gather, I'm seeing 'normal', and current volume is perfectly manageable (it even seems to have dropped-off).

For fun, I knocked-up a little script to scan for UFW blocks and pinpoint the source ip on Google Maps.

Not accurate in all cases of course - but there are some hilarious results.

If nothing else some free data to play with.


----------



## Aldryic C'boas (Nov 18, 2014)

Out of curiosity.. what's the one service preventing you from locking SSH down to key access only?  If it's something like FTP you could always go the SCP/SFTP route.


----------



## stim (Nov 18, 2014)

Aldryic C said:


> Out of curiosity.. what's the one service preventing you from locking SSH down to key access only?  If it's something like FTP you could always go the SCP/SFTP route.


It's a proprietary Windows client. I've tried using Paegent to push the keys to it but as yet no success.

Indeed if there is another way to get windows applications to pull the keys I'd love to know...


----------



## fixidixi (Nov 20, 2014)

@stim:

I dont exactly get it why would you be locked down to an ssh client.

there are a whole bunch of putty forks out there with tabbed option or my personal favourite: puttytray. well i have only 20 connections or so its all well under control i guess it would be too much at a certain point. but with "store in file" option puttytray is pretty portable and thats a huge advantage for me


----------



## Wild1145 (Nov 20, 2014)

Wildcard banning IP's is not often an amazing idea... Often attacks will come from multiple ranges where only maybe 10 IP's are actually doing anything to harm your server... You could change the SSH port to something stupid that people wont guess, but from my understanding there is some software that does require it to be on port 22. Plus you can run into firewall issues on networks if you are not connecting from home (Eg Work)


----------



## stim (Nov 21, 2014)

@fixidixi

No, it's a little more complex than that. In the end I have come up with a simple stopgap solution where I can temporarily enable passwords, then re-disable them after the client connects (it's not a dedicated ssh client, but works over sftp).

Also, I have a very nice and portable set-up already using MultiTabbed Putty and Kitty

@Wild1145

SSH port is already changed. Nobody is getting through the firewall. If they do find my ssh port they are welcome to try.

From the statistics I've gathered so, I am clearly inclined to block all Chinese IP addresses in future.

I've also fired-off a few angry e-mails to hosting providers facilitating the most persistent IPs. Surely they must notice the drain.

The most popular port so far is 9064/TCP by a mile, but mostly via only 4 Chinese ip addresses.

After 2 days, the scores are:

9064/TCP 653 blocks
8118/TCP 288 blocks
23/TCP 233 blocks
43767/UDP 181 blocks
22/TCP 80 blocks
5060/UDP 68 blocks
1433/TCP 54 blocks
443/TCP 54 blocks
8080/TCP 52 blocks
3389/TCP 42 blocks
13814/UDP 40 blocks
3128/TCP 37 blocks
1900/UDP 36 blocks
8088/TCP 26 blocks
3306/TCP 25 blocks
8123/TCP 25 blocks
25/TCP 23 blocks
110/TCP 19 blocks
53/UDP 19 blocks
5900/TCP 17 blocks
8081/TCP 15 blocks
123/UDP 13 blocks
4899/TCP 13 blocks
81/TCP 13 blocks
19/UDP 11 blocks
9000/TCP 11 blocks
135/TCP 10 blocks
21320/TCP 10 blocks
8000/TCP 10 blocks
1080/TCP 9 blocks
21/TCP 9 blocks
83/TCP 8 blocks
8090/TCP 7 blocks
84/TCP 7 blocks
8585/TCP 7 blocks
8888/TCP 7 blocks
9200/TCP 7 blocks
0/TCP 6 blocks


----------



## MartinD (Nov 21, 2014)

That looks about right.

Have you looked at using IPSet for blocks across all your machines?


----------



## stim (Nov 21, 2014)

MartinD said:


> That looks about right.
> 
> Have you looked at using IPSet for blocks across all your machines?


Yeah, I have. And my provider will do it for me if I need. I'm just going to leave to for a while, observe and worry about it when I'm production-ready. At that stage I will probably do a fresh install.

I'll develop my script some more and publish it. It's a nice little eye-opener.

Cheers for all the help and reassurance


----------



## Abdussamad (Nov 21, 2014)

What about whitelisting your IPs? Or using a jump host and whitelisting that alone.

OpenSSH/Cookbook/Proxies and Jump Hosts - Wikibooks, open books for an open world


----------



## MartinD (Nov 21, 2014)

stim said:


> Yeah, I have. And my provider will do it for me if I need. I'm just going to leave to for a while, observe and worry about it when I'm production-ready. At that stage I will probably do a fresh install.
> 
> I'll develop my script some more and publish it. It's a nice little eye-opener.
> 
> Cheers for all the help and reassurance


I've just had a customer (literally in the past 10 minutes) asking me to help with blocking China from their VM for this very reason! You're certainly not alone with the problem!


----------



## stim (Dec 2, 2014)

Hi,

Back again with a related question:

Two of the most scanned ports on my server are:

9064/TCP 3776 blocks
43767/UDP 805 blocks

9064 is constantly being scanned by the same Chinese subnet, accounting for over one third of all connection attempts.

Similarly, I've noticed 43767 getting absolutely hammered since yesterday, but this time from multiple sources.

Both ports are closed, and a Google search suggests that they are generally unassigned.

So why should these ports attract such attention? 

Any thoughts//experiences?

Cheers


----------



## MartinD (Dec 2, 2014)

Who knows with the Chinese. They may have found a few targets that responded on those ports for a certain service so now they're scanning everything/anything to try and get a hit.

I stopped trying to figure out "Why" when it comes to the Chinese and I now settle on "because China".


----------



## Jasson.Pass (Dec 2, 2014)

http://csrc.nist.gov/publications/nistpubs/800-44-ver2/SP800-44v2.pdf


----------



## HalfEatenPie (Dec 2, 2014)

I'm too lazy to check myself, but maybe take a look at this: https://www.us-cert.gov/ncas/bulletins/SB14-335

Could be a recently vulnerability it was testing for.  

But honestly, what MartinD said is right.  "Because China".  

Most of the malicious traffic I get from my servers are also from China.  Second highest though is the United States (lol).


----------



## fixidixi (Dec 2, 2014)

HalfEatenPie said:


> [..]
> 
> Most of the malicious traffic I get from my servers are also from China.  Second highest though is the United States (lol).


 B) Thats a good one..

Thats a lie! There are only angels living in the US! Even the servers (which anyone can rent by the way) are spreading democarcy .


----------



## HalfEatenPie (Dec 2, 2014)

fixidixi said:


> B) Thats a good one..
> 
> Thats a lie! There are only angels living in the US! Even the servers (which anyone can rent by the way) are spreading democarcy .


You know what.  It's not malicious traffic from the US.  It's FREEDOM TRAFFIC.  READY TO LIBERATE MY SERVER IN THE NAME OF DEMOCRACY!

WHERE'S AN ASSAULT RIFLE WHEN YOU NEED IT?  HOO RAH!


----------



## drmike (Dec 3, 2014)

HalfEatenPie said:


> You know what.  It's not malicious traffic from the US.  It's FREEDOM TRAFFIC.  READY TO LIBERATE MY SERVER IN THE NAME OF DEMOCRACY!
> 
> WHERE'S AN ASSAULT RIFLE WHEN YOU NEED IT?  HOO RAH!


Oh, the US can gladly liberate your data with a bunker buster or some heat seeking missiles aimed at your datacenter.


----------



## stim (Dec 3, 2014)

The Yanks don't bother me - even if they are No. 2. The Brits even less.

China is way out in the lead with 63%. What a waste.


----------



## EnveraHost (Dec 7, 2014)

I would suggest whitelisting your IP's, a few remote ones too such as a private VPN just to make sure you still retain access.


----------



## stim (Dec 9, 2014)

EnveraHost said:


> I would suggest whitelisting your IP's, a few remote ones too such as a private VPN just to make sure you still retain access.



This is what I will do eventually.

However just observing has proved useful and informative. Nearly 50% of blocks are from a single Chinese subnet.

I've also been having fun with the geolocation data. I reported some idetifiable misdemeanors to Hosting companies and they all took action, fair play to them.

I'm also pretty sure there is a government facility in St Louis. They don't seem bothered about hiding themsleves. Wankers.


----------



## Serveo (Dec 15, 2014)

MartinD said:


> I stopped trying to figure out "Why" when it comes to the Chinese and I now settle on "because China".


Information is king (-;

@stim why don't you use something like fail2ban?


----------



## rampro (Jan 16, 2015)

Most of the Chinese probes are originating from Chinese made Digital Video Recorders.  Even if these  probes originate from US/UK/Mexico/Thailand/Hong Kong, they are from Chinese made DVRs[SIZE=14.3999996185303px] [/SIZE]installed there. 

[SIZE=14.3999996185303px]In some cases, these probes originate from ADSL Modems. [/SIZE]

[SIZE=14.3999996185303px]The embedded software in these devices has SDK port opened to accept dynamic updates, through which the handlers are making the probes.[/SIZE]

[SIZE=14.3999996185303px]In most cases the equipment owners are innocents who kept opened their devices to the Internet.[/SIZE]

[SIZE=14.3999996185303px]There seems to be a coordinated execution by the firmware creators/exploiters.[/SIZE]


----------



## winnervps (Jan 19, 2015)

In WHM/Cpanel, there is a CSF script that could fetch IP addresses from SPAMcop, etc. (in the lfd.blocklist)

In securing a server, is anybody willing to create such script (at github probably)? or similiar script to fetch such databases and inject into our IPTABLES block list? Thanks *just an idea/or there were already there*


----------



## drmike (Jan 20, 2015)

winnervps said:


> In WHM/Cpanel, there is a CSF script that could fetch IP addresses from SPAMcop, etc. (in the lfd.blocklist)
> 
> In securing a server, is anybody willing to create such script (at github probably)? or similiar script to fetch such databases and inject into our IPTABLES block list? Thanks *just an idea/or there were already there*


What is in the lfd.blocklist for sources ???  Post those if you can.


----------



## souen (Jan 20, 2015)

9064 looks like a common proxy port. Not sure about the other one.


----------

