iptables...I own you!
the_Grinch
Member Posts: 4,165 ■■■■■■■■■■
in Off-Topic
When we first deployed Elasticsearch we had major issues with CentOS. This lead us to using Ubuntu because time was limited and we needed a proof of concept up quickly. It worked well and we got the go ahead to go further. Following training and general browsing we finally found that it was time to expand our cluster. Sadly, we continued to use the Ubuntu box as our master, but I was able to look into the issue and found that the CentOS problems we had were due to iptables. Got the proper rules in place and we were running without issue. Today, needing the box that housed Ubuntu, we decided to offload that node onto a CentOS box thinking we knew how to repair any iptable problems.
Go through reconfiguring ip addresses, adjusting memory settings, and deploying new iptables. Start it up and everything appeared ok, but when we tried to access our web interface we got nothing. Start combing through the logs and we find that the self elected master is denying access to the new CentOS node. Turn off the iptables and everything begins to play nicely. My team mate had to leave and that left me to continue to work on the issue. I kept looking at the rules because I knew it had to be something there causing the problem. Low and behold it was a rules order issue. When we added the ips that could communicate over the proper ports we added saying 4 (thus first ip was rule 1, second was rule 2, etc) and then applied an explicit deny to anyone else trying to talk on that port as rule 5. Since we used various orders of the ip's, on the machines rule "5" came before the allow rule for the new box and it stopped the connection.
Grinch - 1
iptables - 0
Go through reconfiguring ip addresses, adjusting memory settings, and deploying new iptables. Start it up and everything appeared ok, but when we tried to access our web interface we got nothing. Start combing through the logs and we find that the self elected master is denying access to the new CentOS node. Turn off the iptables and everything begins to play nicely. My team mate had to leave and that left me to continue to work on the issue. I kept looking at the rules because I knew it had to be something there causing the problem. Low and behold it was a rules order issue. When we added the ips that could communicate over the proper ports we added saying 4 (thus first ip was rule 1, second was rule 2, etc) and then applied an explicit deny to anyone else trying to talk on that port as rule 5. Since we used various orders of the ip's, on the machines rule "5" came before the allow rule for the new box and it stopped the connection.
Grinch - 1
iptables - 0
WIP:
PHP
Kotlin
Intro to Discrete Math
Programming Languages
Work stuff
PHP
Kotlin
Intro to Discrete Math
Programming Languages
Work stuff
Comments
-
pram
Member Posts: 171
Are you guys not normally linux admins or something, because that is an incredibly mundane issue (no offense) -
the_Grinch
Member Posts: 4,165 ■■■■■■■■■■
Where I work my actual job duties should be auditing only and everything else handle by IT. Need required that we stand up a number of systems ourselves and while I have a more then decent amount linux knowledge I've mainly dealt with Windows servers when I've dealt with servers. Personal accomplishment
and no offense taken. WIP:
PHP
Kotlin
Intro to Discrete Math
Programming Languages
Work stuff -
the_Grinch
Member Posts: 4,165 ■■■■■■■■■■
Once everything is in place we need to limit what nodes have access to these machines. Elasticsearch has no security what's so ever and we want to prevent unauthorized parties from performing queries on it.WIP:
PHP
Kotlin
Intro to Discrete Math
Programming Languages
Work stuff -
ccnxjr
Member Posts: 304 ■■■□□□□□□□
Don't forget to run:service iptables save
(I understand server admins come in many varieties, it's easy to forget that there are whole other flavors of Sys Admins!) -
phoeneous
Member Posts: 2,333 ■■■■■■■□□□
Good thing it wasnt an selinux issue, those are fun. By fun I mean not really fun. -
LinuxNerd
Member Posts: 83 ■■□□□□□□□□
Why do you need IP tables on the local machine for?
CentOS has Netfilter enabled by default and blocks all ports on a default install except SSH. Ubuntu doesn't have Netfilter enabled by default.
I love Iptables and they can add a nice HIDS layer to a machine but must be configured correctly. -
the_Grinch
Member Posts: 4,165 ■■■■■■■■■■
Yup I saved all the rules on each of the box and made sure I tested by stopping then starting them again. Thanks all!!WIP:
PHP
Kotlin
Intro to Discrete Math
Programming Languages
Work stuff -
UnixGuy
Mod Posts: 4,564 Mod
@LinuxNerd:
Netfilter is enabled by default in CentOS that's true but the default configuration is useless because the default Policy is ACCEPT, so the rules don't block anything really.
I agree, IPtables adds a nice layer of security, but with some application the initial configuration can be painful, but once it's configured then we can forget about it. -
LinuxNerd
Member Posts: 83 ■■□□□□□□□□
@LinuxNerd:
Netfilter is enabled by default in CentOS that's true but the default configuration is useless because the default Policy is ACCEPT, so the rules don't block anything really.
Don't have time to debate your inaccuracies, too busy developing serious code. -
tprice5
Member Posts: 770
Don't have time to debate your inaccuracies, too busy developing serious code.
Wow... Going to check back and wait for his rep to go red.Certification To-Do: CEH [ ], CHFI [ ], NCSA [ ], E10-001 [ ], 70-413 [ ], 70-414 [ ]
WGU MSISA
Start Date: 10/01/2014 | Complete Date: ASAP
All Courses: LOT2, LYT2 , UVC2, ORA1, VUT2, VLT2 , FNV2 , TFT2 , JIT2 , FMV2, FXT2 , LQT2 -
UnixGuy
Mod Posts: 4,564 Mod
Yes you are right, it's inaccurate to say that they don't block anything, because they do. It's more accurate to say that they block stuff but the default policy is to ACCEPT (that's what I meant).
I should have elaborated more, as a practice I always wipe the configuration out, change the default policy to DROP, and allow the things that are needed.
Good luck with your serious code.
