Houston We Don't Have a Problem!

the_Grinch

At work we undertook a major project using Elasticsearch and it has been bumpy to say the least. An issue with the approval for training stopped us from going in April and we had started implementing it in March. The back and forth lead to us finally going for training in June when at that point we had to question how much we would have taken away. In that time it has been a constant battle of keeping the system "up". What we use it for is actually not as common as we would have liked (out of 20 people in training only one other person was utilizing in a manner that was close to what we are doing). When I say keeping it "up" it was really a matter of data no longer showing up in the web interface. We had a span of about a seven weeks where we never had to restart the service. Initially we thought it was because we hadn't installed it (just compiled and ran from source). We install it and we got our four weeks of solid no restarting services time.

By the time we went to training we were hitting the restart services button on a daily basis. At the training a lot was over our heads, but we got access to the creators of Elasticsearch and to best practices (along with troubleshooting methods). Push come to shoves and today (after fiddling with it for an hour last night pretty late) I've hit over 12 hours with no service restart. Hopefully not counting my chickens before they hatch, but I think we got it!

ccnxjr

By "installed it", I'm guessing you used an rpm or deb package?
(as opposed to an ./install.sh script or some configmatron )

Any theories on which fiddling kept it going strong?

the_Grinch

Correct we installed the rpms for it. There are a couple of things to consider when setting things up with Elasticsearch:

IPTables - they'll wreck your life so be sure to open up 9200 to 9400 (Elasticsearch uses 9200 to 9299 for itself and 9300 to 9399 for transport between nodes in the cluster)

RAM - You can tell it how much ram you want it to use and they recommend that you use half of what you have on your box - you can set this in your heap size settings when I get home I can post where in CentOS this is located

bootstrap.mlockall - In your Elasticsearch config set this to true - JVM swamps memory locations and it causes issues thus this will lock you in (should dedicate your servers to Elasticsearch so no issues)

ulimit -u unlimited - run this as root on your boxes

I believe if you follow all of this if you were having issues you should be good to go.

From any node on your cluster type this command:
curl -XGET 'http://localhost:9200/_cluster/health?pretty=true'

That will give you the status of your cluster - RED is bad. Usually this will happen if shards are unassigned and if that is the case you'll have to hunt them down. Then you can assign them to the correct node and be good to go.

darkerosxx

Is this a Splunk competitor, do you know? Looks like it from what I can tell.

the_Grinch

Yup they are a competitor with Splunk. Splunk's frontend is a lot better, but after what we showed a vendor who was using Splunk what we were doing they are now actively looking to move to Elasticsearch. Splunk can become very expensive depending on your environment. Elasticsearch is open source and free (at least currently).

the_Grinch

As promised, if you installed via the rpm in Centos 6.5 go to the following directory to change the default heap size:

/etc/sysconfig/elasticsearch

The default is 1g and it is commented out, so uncomment it and put half the amount of ram you have in the machine: ex 8g for 8 gigabytes

Also it should be noted that you should not go above 30 gigs of ram dedicated to Elasticsearch. A bug in the JVM won't allow you to go above 32 gigs and even that causes issues so 30 gigs is the max.

the_Grinch

24 hours and still working like a champ! Finally nipped it in the butt I think!