Security is indeed a process

lordy

Dear all,

I just wanted to share a little story/discovery with you that I just made.

A few years ago I set up two servers for a project/web-shop that a friend of mine was starting. He was a PHP guy so all my alarms went off when he said that he wanted to take care of the servers. Being friends since school-time I offered to install and secure them so that he could focus on the coding. That was back when Debian 4.0 was new We had it all, GRsecurity-kernel, chrooted-Webserver, you name it.

About two years ago he sold the project to some other guy I don't really know who took over the whole thing. That's when I stopped caring about the boxes, they were no longer my business. When that guy came into financial trouble and asked me if I could somehow lower the server-costs I virtualized the two boxes onto one new host which saved him a bunch of money. Don't ask me why but I somehow did that for free although it was quite some work.

Fast forward to today: He migrated the shop to a managed server and last weekend we also migrated his email service. Checking out the VMs one last time before shutting them down for good I wonder why the load on the web box is 1.00 although it is no longer in production. Turns out somebody root'ed it with an Exim exploit (that's why I usually install qmail !). It was also talking to some random IRC server and probably trying to break some hashes or mining bitcoins. Who ever did this didn't put much effort into hiding his trails. The last log shows some german DSL/dial-up IP, there is a second sshd running on port 50000-something and "ps" and "netstat" seem to be fully functional (I didn't bother to check the signatures).

Morale of the story: It doesn't matter how secure your system is when you deploy it. If you put zero effort into keeping it secure you will be caught with your pants down one day.

docrice

Yup, security is definitely a process, not an end-state. Maintenance is the heavy-lifting when dealing with security. I'm surprised you didn't configure the forward firewall to block egress traffic, but then again I guess you need to send mail outbound and perhaps the attackers were able to download their tools through that avenue.

instant000

Did you inform the guy of the problem with his servers, so that he would be sure to monitor the security of them in the future?

Bl8ckr0uter

Awesome story. +rep

lordy

instant000 wrote: »

Did you inform the guy of the problem with his servers, so that he would be sure to monitor the security of them in the future?

Of course. Luckily all his services are now managed so that will now be somebody elses problem

it_consultant

lordy wrote: »

Dear all,

I just wanted to share a little story/discovery with you that I just made.

A few years ago I set up two servers for a project/web-shop that a friend of mine was starting. He was a PHP guy so all my alarms went off when he said that he wanted to take care of the servers. Being friends since school-time I offered to install and secure them so that he could focus on the coding. That was back when Debian 4.0 was new We had it all, GRsecurity-kernel, chrooted-Webserver, you name it.

About two years ago he sold the project to some other guy I don't really know who took over the whole thing. That's when I stopped caring about the boxes, they were no longer my business. When that guy came into financial trouble and asked me if I could somehow lower the server-costs I virtualized the two boxes onto one new host which saved him a bunch of money. Don't ask me why but I somehow did that for free although it was quite some work.

Fast forward to today: He migrated the shop to a managed server and last weekend we also migrated his email service. Checking out the VMs one last time before shutting them down for good I wonder why the load on the web box is 1.00 although it is no longer in production. Turns out somebody root'ed it with an Exim exploit (that's why I usually install qmail !). It was also talking to some random IRC server and probably trying to break some hashes or mining bitcoins. Who ever did this didn't put much effort into hiding his trails. The last log shows some german DSL/dial-up IP, there is a second sshd running on port 50000-something and "ps" and "netstat" seem to be fully functional (I didn't bother to check the signatures).

Morale of the story: It doesn't matter how secure your system is when you deploy it. If you put zero effort into keeping it secure you will be caught with your pants down one day.

I had the police arrive at a client of mine because one of our servers was using mIRC to try and sell credit cards online. Someone from Romania guessed their domain admin password (it was stupidly simple and was not changed often enough) and RDP'd into a test platform server that was normally accessed by a 3rd party vendor. That'll teach you....

chrisone

Awesome story! i need to learn all forms of security threats like this one. Now that R&S/Design is done, Security is my journey for the next 2 years. CCNA Security, CCNP Security, CEH, OSCP, GPEN.

I enjoy reading material like this