iptables...I own you!

the_Grinchthe_Grinch Member Posts: 4,165 ■■■■■■■■■■
When we first deployed Elasticsearch we had major issues with CentOS. This lead us to using Ubuntu because time was limited and we needed a proof of concept up quickly. It worked well and we got the go ahead to go further. Following training and general browsing we finally found that it was time to expand our cluster. Sadly, we continued to use the Ubuntu box as our master, but I was able to look into the issue and found that the CentOS problems we had were due to iptables. Got the proper rules in place and we were running without issue. Today, needing the box that housed Ubuntu, we decided to offload that node onto a CentOS box thinking we knew how to repair any iptable problems.

Go through reconfiguring ip addresses, adjusting memory settings, and deploying new iptables. Start it up and everything appeared ok, but when we tried to access our web interface we got nothing. Start combing through the logs and we find that the self elected master is denying access to the new CentOS node. Turn off the iptables and everything begins to play nicely. My team mate had to leave and that left me to continue to work on the issue. I kept looking at the rules because I knew it had to be something there causing the problem. Low and behold it was a rules order issue. When we added the ips that could communicate over the proper ports we added saying 4 (thus first ip was rule 1, second was rule 2, etc) and then applied an explicit deny to anyone else trying to talk on that port as rule 5. Since we used various orders of the ip's, on the machines rule "5" came before the allow rule for the new box and it stopped the connection.

Grinch - 1
iptables - 0
WIP:
PHP
Kotlin
Intro to Discrete Math
Programming Languages
Work stuff

Comments

  • prampram Member Posts: 171
    Are you guys not normally linux admins or something, because that is an incredibly mundane issue (no offense)
  • the_Grinchthe_Grinch Member Posts: 4,165 ■■■■■■■■■■
    Where I work my actual job duties should be auditing only and everything else handle by IT. Need required that we stand up a number of systems ourselves and while I have a more then decent amount linux knowledge I've mainly dealt with Windows servers when I've dealt with servers. Personal accomplishment ;) and no offense taken.
    WIP:
    PHP
    Kotlin
    Intro to Discrete Math
    Programming Languages
    Work stuff
  • UnixGuyUnixGuy Are we having fun yet? Mod Posts: 4,305 Mod
    Why do you need IP tables on the local machine for?
    Certs: GPEN, GCFA, CISM, CRISC, RHCE
    In Progress: MBA
  • the_Grinchthe_Grinch Member Posts: 4,165 ■■■■■■■■■■
    Once everything is in place we need to limit what nodes have access to these machines. Elasticsearch has no security what's so ever and we want to prevent unauthorized parties from performing queries on it.
    WIP:
    PHP
    Kotlin
    Intro to Discrete Math
    Programming Languages
    Work stuff
  • ccnxjrccnxjr Member Posts: 304
    Don't forget to run:
    service iptables save
    


    (I understand server admins come in many varieties, it's easy to forget that there are whole other flavors of Sys Admins!)
  • phoeneousphoeneous Go ping yourself... Member Posts: 2,333 ■■■■■■■□□□
    Good thing it wasnt an selinux issue, those are fun. By fun I mean not really fun.
  • LinuxNerdLinuxNerd Member Posts: 83 ■■□□□□□□□□
    UnixGuy wrote: »
    Why do you need IP tables on the local machine for?

    CentOS has Netfilter enabled by default and blocks all ports on a default install except SSH. Ubuntu doesn't have Netfilter enabled by default.

    I love Iptables and they can add a nice HIDS layer to a machine but must be configured correctly.
  • the_Grinchthe_Grinch Member Posts: 4,165 ■■■■■■■■■■
    Yup I saved all the rules on each of the box and made sure I tested by stopping then starting them again. Thanks all!!
    WIP:
    PHP
    Kotlin
    Intro to Discrete Math
    Programming Languages
    Work stuff
  • UnixGuyUnixGuy Are we having fun yet? Mod Posts: 4,305 Mod
    @LinuxNerd:

    Netfilter is enabled by default in CentOS that's true but the default configuration is useless because the default Policy is ACCEPT, so the rules don't block anything really.

    I agree, IPtables adds a nice layer of security, but with some application the initial configuration can be painful, but once it's configured then we can forget about it.
    Certs: GPEN, GCFA, CISM, CRISC, RHCE
    In Progress: MBA
  • LinuxNerdLinuxNerd Member Posts: 83 ■■□□□□□□□□
    UnixGuy wrote: »
    @LinuxNerd:

    Netfilter is enabled by default in CentOS that's true but the default configuration is useless because the default Policy is ACCEPT, so the rules don't block anything really.

    Don't have time to debate your inaccuracies, too busy developing serious code.
  • tprice5tprice5 Member Posts: 770
    LinuxNerd wrote: »
    Don't have time to debate your inaccuracies, too busy developing serious code.

    Wow... Going to check back and wait for his rep to go red.
    Certification To-Do: CEH [ ], CHFI [ ], NCSA [ ], E10-001 [ ], 70-413 [ ], 70-414 [ ]
    WGU MSISA
    Start Date: 10/01/2014 | Complete Date: ASAP
    All Courses: LOT2, LYT2 , UVC2, ORA1, VUT2, VLT2 , FNV2 , TFT2 , JIT2 , FMV2, FXT2 , LQT2
  • UnixGuyUnixGuy Are we having fun yet? Mod Posts: 4,305 Mod
    Yes you are right, it's inaccurate to say that they don't block anything, because they do. It's more accurate to say that they block stuff but the default policy is to ACCEPT (that's what I meant).

    I should have elaborated more, as a practice I always wipe the configuration out, change the default policy to DROP, and allow the things that are needed.

    Good luck with your serious code.
    Certs: GPEN, GCFA, CISM, CRISC, RHCE
    In Progress: MBA
Sign In or Register to comment.