Two weeks ago my server was used to attempt break-ins of at least three other servers. Before this happened I was totally unaware of what was needed to secure a web server. I'm still not close to being a security expert but I have learned quite a bit in the past few days. In this post I hope to provide the uninformed or newby server administrator with information and tools to secure their server against hackers.
My experience began on a Friday evening when I checked my inbox and found several emails from other server administrators, suggesting that my server had been compromised and was being used to attempt break-ins of their systems.
The first thing I did was log into WHM and checked the Show Current CPU Usage link. I noticed that the CPU was pegged at 100% by several Nobody users. I should of taken the time to record exactly what processes were being used but I immediately killed all Nobody processes. This action brought the server back to normal operation.
I honestly did not know how to proceed. I was under the assumption that since I had been operating this server for several years without incident, that I had a fairly secure server. I started with Googling Apache security topics and visited the discussion forums hosted by the company where I lease my dedicated server.
The two applications that I consistently read about were APF (Advanced Policy Firewall) and BFD (Brute Force Detection). I've always used a firewall on my personal PC but it never occurred to me that I would need a firewall for my server.
I've never installed programs on the server before so this was a new learning process. Luckily I found a great website that took me step by step through the process.
Once I got AFP and BFD installed I began searching for the culprit. I learned that most rootkits are stored and run from /tmp folders, so I ran the following commands from SSH as a root user.
CODE:
-
-
# find /dev -type f
-
# find /tmp -type f | less
-
# find /var/tmp -type f | less
-
When I got to the /var/tmp/ directory I found what I was looking for, a suspicious directory called /bebe/. In this directory were several files beginning with an IP subset.pscan.22. I'm assuming the IP subsets were the IP addresses my system was scanning and the port was 22.
I zipped up the files, downloaded a copy and then deleted the directory.
From what I have read, once a rootkit is uploaded to your system the hard drive should be formatted followed by a fresh install of the OS. I took several additional steps and I feel fairly confident that I'm now running a clean system. I'll follow up with another post on what additional steps I've taken, and tell you what my daily security routine involves.