iptables...I own you!

the_Grinch

When we first deployed Elasticsearch we had major issues with CentOS. This lead us to using Ubuntu because time was limited and we needed a proof of concept up quickly. It worked well and we got the go ahead to go further. Following training and general browsing we finally found that it was time to expand our cluster. Sadly, we continued to use the Ubuntu box as our master, but I was able to look into the issue and found that the CentOS problems we had were due to iptables. Got the proper rules in place and we were running without issue. Today, needing the box that housed Ubuntu, we decided to offload that node onto a CentOS box thinking we knew how to repair any iptable problems.

Go through reconfiguring ip addresses, adjusting memory settings, and deploying new iptables. Start it up and everything appeared ok, but when we tried to access our web interface we got nothing. Start combing through the logs and we find that the self elected master is denying access to the new CentOS node. Turn off the iptables and everything begins to play nicely. My team mate had to leave and that left me to continue to work on the issue. I kept looking at the rules because I knew it had to be something there causing the problem. Low and behold it was a rules order issue. When we added the ips that could communicate over the proper ports we added saying 4 (thus first ip was rule 1, second was rule 2, etc) and then applied an explicit deny to anyone else trying to talk on that port as rule 5. Since we used various orders of the ip's, on the machines rule "5" came before the allow rule for the new box and it stopped the connection.

Grinch - 1
iptables - 0

pram

Are you guys not normally linux admins or something, because that is an incredibly mundane issue (no offense)

the_Grinch

Where I work my actual job duties should be auditing only and everything else handle by IT. Need required that we stand up a number of systems ourselves and while I have a more then decent amount linux knowledge I've mainly dealt with Windows servers when I've dealt with servers. Personal accomplishment and no offense taken.

UnixGuy

Why do you need IP tables on the local machine for?

the_Grinch

Once everything is in place we need to limit what nodes have access to these machines. Elasticsearch has no security what's so ever and we want to prevent unauthorized parties from performing queries on it.

ccnxjr

Don't forget to run:

service iptables save

(I understand server admins come in many varieties, it's easy to forget that there are whole other flavors of Sys Admins!)

phoeneous

Good thing it wasnt an selinux issue, those are fun. By fun I mean not really fun.

LinuxNerd

UnixGuy wrote: »

Why do you need IP tables on the local machine for?

CentOS has Netfilter enabled by default and blocks all ports on a default install except SSH. Ubuntu doesn't have Netfilter enabled by default.

I love Iptables and they can add a nice HIDS layer to a machine but must be configured correctly.

the_Grinch

Yup I saved all the rules on each of the box and made sure I tested by stopping then starting them again. Thanks all!!

UnixGuy

@LinuxNerd:

Netfilter is enabled by default in CentOS that's true but the default configuration is useless because the default Policy is ACCEPT, so the rules don't block anything really.

I agree, IPtables adds a nice layer of security, but with some application the initial configuration can be painful, but once it's configured then we can forget about it.

LinuxNerd

UnixGuy wrote: »

@LinuxNerd:

Netfilter is enabled by default in CentOS that's true but the default configuration is useless because the default Policy is ACCEPT, so the rules don't block anything really.

Don't have time to debate your inaccuracies, too busy developing serious code.

tprice5

LinuxNerd wrote: »

Don't have time to debate your inaccuracies, too busy developing serious code.

Wow... Going to check back and wait for his rep to go red.

UnixGuy

Yes you are right, it's inaccurate to say that they don't block anything, because they do. It's more accurate to say that they block stuff but the default policy is to ACCEPT (that's what I meant).

I should have elaborated more, as a practice I always wipe the configuration out, change the default policy to DROP, and allow the things that are needed.

Good luck with your serious code.