Looking for an alternative to Splunk.

msteinhilber

As part of trying to make our environment more manageable, secure, and available - I've had a desire to consolidate logs to a centralized location for identifying issues, reporting, trending, etc. Splunk is pretty slick, but frankly our budget would never accommodate purchasing it and we would go through the 500mb/day volume so the free version won't cut it.

Has anybody worked with anything somewhat similar? To be honest, anything is better than what we're doing now - which is essentially ignoring all log files unless we are aware of some sort of problem. It doesn't have to be as searchable as Splunk is - which seems to be one of it's strongest points. Simply consolidating logs to central point and checking for particular logged events and alerting, archival, and perhaps some graphing ability would be nice but not required. Alerting on particular events and just archiving logs in case we need to reference past logs in the future are the two primary features I'd like.

So far I'm leaning towards setting up a simple Linux server with rsyslog and LogAnalyzer (formerly phpLogCon) but was wondering if any of the brilliant minds on TE had any other suggestions before I get too far involved going this route

xpolog

Have a look on XpoLog at XpoLog Log Management and Log Analysis Platform
It is a log analysis platform with search engine, log management, log viewer, reports, automatic pattern analysis, semantic analysis,log analysis, reports and more.

msteinhilber

xpolog wrote: »

Have a look on XpoLog at XpoLog Log Management and Log Analysis Platform
It is a log analysis platform with search engine, log management, log viewer, reports, automatic pattern analysis, semantic analysis,log analysis, reports and more.

Anybody else have any suggestions that aren't from a company seeking to tout their own product by spamming a forum?

ccosk

take a look at f-deets (google), supports generic syslog, windows client, windows and linux server...
/gjb

Forsaken_GA

I just send everything to a common syslog server, and then have sec (Simple Event Correllator) watch the log files. You can define the events you want to watch for in the configuration file, and then have it react based on those. It can call pretty much anything external you want, so if you want it to send SNMP traps, it'll do that. You want it to email you, it'll do that. You want it to rm -rf /, it can do that too, but I don't recommend it

If you're comfortable with regex, this will probably be the easiest and most cost effective method, assuming everything you want to monitor can actually send to syslog, though you don't necessarily need to deploy it on the syslog server. You could, for example, deploy it on a mysql server, and have it watch mysql logs for things like database corruptions and the like, and react based on that. It all depends on what you want to do, but there's plenty of good solutions out there if you deploy a little ingenuity

Right now, I use it to watch for things like catastrophic layer 2 events (Loop Guard getting triggered), interfaces going down, routing protocol convergence, and for ARP messages that constantly change the MAC for the same IP.

I use syslog-ng for the syslog server, and just let logrotate handle local archival duties, and use a weekly rsync script to copy log archives about to be deleted to a permanent log retention server for long term storage. Hasn't cost the company a dime beyond a server for the setup, and my salary.

terryfera

I found an open source product that offers enough of the features to be interesting while I was searching for the same thing.

Graylog2 - Free open source self-hosted log management and exception tracking

I haven't had a chance to really put it through it's paces but it looks like a good alternative to splunk.

Forsaken_GA

Hrm, Graylog looks interesting, could be useful as a front-end for correlation, but it's notification options kind of suck. Email alerting only, and it's dependent on a cron job to check for exceeded thresholds, so there's some latency built in.

It's something I'll probably play with, but I'd have a hard time deploying it in production. Far too many moving parts on the backend, and I really hate points of failure in my critical systems (and I consider monitoring a *very* critical system)

JDMurray

Here's a listing of log file viewers that support Apache log files, as I assume most log file viewers would: log4j - Wikipedia, the free encyclopedia